00010000:00010000:2.0:1788571846.547033:0:59055:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f600/0x1e260726fbbbfbbe lrc: 4/1,0 mode: --/CR res: [0x2000061c1:0x17:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 59055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571846.547055:0:59055:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f600/0x1e260726fbbbfbbe lrc: 4/1,0 mode: --/CR res: [0x2000061c1:0x17:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 59055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571846.547061:0:59055:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f600/0x1e260726fbbbfbbe lrc: 3/0,0 mode: --/CR res: [0x2000061c1:0x17:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 59055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571846.547070:0:59055:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f600/0x1e260726fbbbfbbe lrc: 4/0,0 mode: --/CR res: [0x2000061c1:0x17:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 59055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571846.547077:0:59055:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d41251f600 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:2.0:1788571846.547082:0:59055:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f600/0x1e260726fbbbfbbe lrc: 4/0,0 mode: --/CR res: [0x2000061c1:0x17:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 59055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571846.547089:0:59055:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f600/0x1e260726fbbbfbbe lrc: 4/0,0 mode: --/CR res: [0x2000061c1:0x17:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 59055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571846.547096:0:59055:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f600/0x1e260726fbbbfbbe lrc: 5/0,0 mode: --/CR res: [0x2000061c1:0x17:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x20e400000000 pid: 59055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571846.547102:0:59055:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788571846.547109:0:59055:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f600/0x1e260726fbbbfbbe lrc: 3/0,0 mode: --/CR res: [0x2000061c1:0x17:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 59055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571846.547124:0:59055:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f600/0x1e260726fbbbfbbe lrc: 0/0,0 mode: --/CR res: [0x2000061c1:0x17:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 59055 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788571846.547143:0:59055:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff94d408c61500 x1875450488285056/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 584/584 e 0 to 0 dl 1788571862 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788571846.547157:0:59055:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2000061c1:0x17:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1788571846.547193:0:59055:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 30 res 8589959617/23 (0 0) 00000080:00200000:2.0:1788571846.547283:0:59055:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788571846.547297:0:59055:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788571846.547306:0:59055:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f800/0x1e260726fbbbfb32 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbb79 expref: -99 pid: 59035 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:2.0:1788571846.547791:0:59055:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f800/0x1e260726fbbbfb32 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbb79 expref: -99 pid: 59035 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200000:2.0:1788571846.547801:0:59055:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788571846.547808:0:59055:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788571846.547812:0:59055:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbbfb32 00000080:00010000:2.0:1788571846.547818:0:59055:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfb32 from it ffffb68a42087b58 00010000:00010000:2.0:1788571846.547821:0:59055:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f800/0x1e260726fbbbfb32 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dbb79 expref: -99 pid: 59035 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:2.0:1788571846.547829:0:59055:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f800/0x1e260726fbbbfb32 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbb79 expref: -99 pid: 59035 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00200000:2.0:1788571846.547844:0:59055:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788571846.547853:0:59055:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d118.replay-single, flags=13 00000080:00200008:2.0:1788571846.547864:0:59055:0:(file.c:6285:ll_getattr_dentry()) START file d118.replay-single:[0x2000061c1:0x17:0x0](ffff94d4102242c8), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788571846.547868:0:59055:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x2000061c1:0x17:0x0](ffff94d4102242c8),name=d118.replay-single 00000002:00010000:2.0:1788571846.547882:0:59055:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000061c1:0x17:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788571846.547889:0:59055:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 30 res 8589959617/23 (0 0) 00010000:00010000:2.0:1788571846.547937:0:59055:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbc5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 59055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571846.547945:0:59055:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbc5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 59055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571846.547955:0:59055:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbc5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 59055 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788571846.547976:0:59055:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:59055:1875450488285184:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788571846.548303:0:59055:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4128ff180 going to sleep for 11 seconds 00000100:00100000:2.0:1788571846.556850:0:59055:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:59055:1875450488285184:192.168.202.125@tcp:101:lfs.0 00010000:00010000:2.0:1788571846.556880:0:59055:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbc5 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbda2 expref: -99 pid: 59055 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571846.556898:0:59055:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbc5 lrc: 4/1,0 mode: --/PR res: [0x2000061c1:0x17:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbda2 expref: -99 pid: 59055 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571846.556906:0:59055:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbc5 lrc: 4/1,0 mode: --/PR res: [0x2000061c1:0x17:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbda2 expref: -99 pid: 59055 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571846.557050:0:59055:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbc5 lrc: 4/1,0 mode: PR/PR res: [0x2000061c1:0x17:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dbda2 expref: -99 pid: 59055 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571846.557062:0:59055:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbc5 lrc: 4/1,0 mode: PR/PR res: [0x2000061c1:0x17:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbda2 expref: -99 pid: 59055 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571846.557077:0:59055:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbc5 lrc: 5/2,0 mode: PR/PR res: [0x2000061c1:0x17:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dbda2 expref: -99 pid: 59055 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571846.557085:0:59055:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbc5 lrc: 5/2,0 mode: PR/PR res: [0x2000061c1:0x17:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dbda2 expref: -99 pid: 59055 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571846.557098:0:59055:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbc5 lrc: 4/1,0 mode: PR/PR res: [0x2000061c1:0x17:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dbda2 expref: -99 pid: 59055 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788571846.557111:0:59055:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408c61500 x1875450488285184/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/584 e 0 to 0 dl 1788571862 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788571846.557123:0:59055:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbc5 lrc: 3/1,0 mode: PR/PR res: [0x2000061c1:0x17:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbda2 expref: -99 pid: 59055 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571846.557145:0:59055:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589959617/23 (0 0) 00000080:00200000:2.0:1788571846.557152:0:59055:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788571846.557174:0:59055:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x2000061c1:0x17:0x0], updating i_size 4096 00000080:00010000:2.0:1788571846.557193:0:59055:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000061c1:0x17:0x0](ffff94d4102242c8) for lock 0x1e260726fbbbfbc5 00000080:00010000:2.0:1788571846.557241:0:59055:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000061c1:0x17:0x0](ffff94d4102242c8) 00000080:00010000:2.0:1788571846.557245:0:59055:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfbc5 from it ffffb68a42087cb8 00010000:00010000:2.0:1788571846.557248:0:59055:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbc5 lrc: 3/1,0 mode: PR/PR res: [0x2000061c1:0x17:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dbda2 expref: -99 pid: 59055 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571846.557257:0:59055:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbc5 lrc: 2/0,0 mode: PR/PR res: [0x2000061c1:0x17:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbda2 expref: -99 pid: 59055 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788571846.558725:0:59055:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788571846.558734:0:59055:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788571846.558755:0:59055:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788571846.558770:0:59055:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f800/0x1e260726fbbbfb32 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbb79 expref: -99 pid: 59035 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:2.0:1788571846.558789:0:59055:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f800/0x1e260726fbbbfb32 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbb79 expref: -99 pid: 59035 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00200000:2.0:1788571846.558797:0:59055:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788571846.558804:0:59055:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788571846.558808:0:59055:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbbfb32 00000080:00010000:2.0:1788571846.558815:0:59055:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfb32 from it ffffb68a42087cb8 00010000:00010000:2.0:1788571846.558818:0:59055:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f800/0x1e260726fbbbfb32 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dbb79 expref: -99 pid: 59035 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:2.0:1788571846.558826:0:59055:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f800/0x1e260726fbbbfb32 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbb79 expref: -99 pid: 59035 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000080:00200000:2.0:1788571846.559096:0:59055:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788571846.559106:0:59055:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788571846.559115:0:59055:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f800/0x1e260726fbbbfb32 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbb79 expref: -99 pid: 59035 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:2.0:1788571846.559570:0:59055:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f800/0x1e260726fbbbfb32 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbb79 expref: -99 pid: 59035 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000080:00200000:2.0:1788571846.559579:0:59055:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788571846.559585:0:59055:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788571846.559589:0:59055:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbbfb32 00000080:00010000:2.0:1788571846.559594:0:59055:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfb32 from it ffffb68a42087bf8 00010000:00010000:2.0:1788571846.559598:0:59055:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f800/0x1e260726fbbbfb32 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dbb79 expref: -99 pid: 59035 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:2.0:1788571846.559606:0:59055:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f800/0x1e260726fbbbfb32 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbb79 expref: -99 pid: 59035 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000080:00200000:2.0:1788571846.559618:0:59055:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788571846.559625:0:59055:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d118.replay-single, flags=267 00000080:00200000:2.0:1788571846.559632:0:59055:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x2000061c1:0x17:0x0](ffff94d4102242c8), inode mode 41ed mask 44 00000080:00200008:2.0:1788571846.559639:0:59055:0:(file.c:1090:ll_file_open()) START file d118.replay-single:[0x2000061c1:0x17:0x0](ffff94d4102242c8), flags 300000 00000080:00200080:2.0:1788571846.559664:0:59055:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x2000061c1:0x17:0x0](ffff94d4102242c8) cmd=c03066f0 arg=7ffe39d64a50 00000080:00200000:2.0:1788571846.559679:0:59055:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x2000061c1:0x17:0x0](ffff94d4102242c8) name=striped_dir stripe_offset=-1 stripe_count=2, hash_type=0 00000020:00100000:2.0:1788571846.559742:0:59055:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788571846.559755:0:59055:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c61500 x1875450488285440/t0(0) o36->lustre-MDT0001-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788571846.559779:0:59055:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:59055:1875450488285440:192.168.202.125@tcp:36:lfs.0 00000100:00100000:2.0:1788571846.559874:0:59055:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4128ff180 going to sleep for 11 seconds 00000100:00100000:2.0:1788571846.566724:0:42440:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875450511100800 00000100:00100000:2.0:1788571846.566733:0:42440:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875450511100800 00000100:00100000:2.0:1788571846.566788:0:42440:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.125@tcp, seq: 46 00000100:00100000:2.0:1788571846.566798:0:42440:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff94d408c54e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:9bf287c0-f582-4b83-84c1-15e766218620+5:8443:x1875450511100800:12345-192.168.202.125@tcp:104: 00010000:00010000:2.0:1788571846.566815:0:42440:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbc5 lrc: 2/0,0 mode: PR/PR res: [0x2000061c1:0x17:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xec901abfeb4dbda2 expref: -99 pid: 59055 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788571846.566893:0:42440:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788571846.566900:0:42440:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff94d408c54e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:9bf287c0-f582-4b83-84c1-15e766218620+5:8443:x1875450511100800:12345-192.168.202.125@tcp:104: Request processed in 104us (275us total) trans 0 rc 0/0 00000100:00100000:2.0:1788571846.566908:0:42440:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.125@tcp, seq: 46 00000100:00100000:2.0:1788571846.638703:0:59055:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:59055:1875450488285440:192.168.202.125@tcp:36:lfs.0 00000080:00200008:2.0:1788571846.638824:0:59055:0:(file.c:479:ll_file_release()) START file d118.replay-single:[0x2000061c1:0x17:0x0](ffff94d4102242c8), flags 300000 00010000:00010000:2.0:1788571846.638857:0:59055:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481f200/0x1e260726fbbbfb71 lrc: 2/0,0 mode: CR/CR res: [0x2000061c1:0x17:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbc4b expref: -99 pid: 59041 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:2.0:1788571848.595685:0:59073:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788571848.595702:0:59073:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788571848.595725:0:59073:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788571848.595742:0:59073:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f800/0x1e260726fbbbfb32 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbb79 expref: -99 pid: 59035 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:2.0:1788571848.595757:0:59073:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f800/0x1e260726fbbbfb32 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbb79 expref: -99 pid: 59035 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000080:00200000:2.0:1788571848.595765:0:59073:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788571848.595773:0:59073:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788571848.595776:0:59073:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbbfb32 00000080:00010000:2.0:1788571848.595784:0:59073:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfb32 from it ffffb68a41f77cb8 00010000:00010000:2.0:1788571848.595788:0:59073:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f800/0x1e260726fbbbfb32 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dbb79 expref: -99 pid: 59035 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:2.0:1788571848.595797:0:59073:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f800/0x1e260726fbbbfb32 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbb79 expref: -99 pid: 59035 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000080:00200008:2.0:1788571848.596544:0:59073:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788571848.596551:0:59073:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788571848.596561:0:59073:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788571848.596570:0:59073:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f800/0x1e260726fbbbfb32 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbb79 expref: -99 pid: 59035 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:2.0:1788571848.596579:0:59073:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f800/0x1e260726fbbbfb32 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbb79 expref: -99 pid: 59035 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000080:00200000:2.0:1788571848.596585:0:59073:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788571848.596590:0:59073:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788571848.596593:0:59073:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbbfb32 00000080:00010000:2.0:1788571848.596597:0:59073:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfb32 from it ffffb68a41f77cb8 00010000:00010000:2.0:1788571848.596600:0:59073:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f800/0x1e260726fbbbfb32 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dbb79 expref: -99 pid: 59035 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:2.0:1788571848.596607:0:59073:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f800/0x1e260726fbbbfb32 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbb79 expref: -99 pid: 59035 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000080:00200000:2.0:1788571848.596706:0:59073:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788571848.596715:0:59073:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788571848.596722:0:59073:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f800/0x1e260726fbbbfb32 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbb79 expref: -99 pid: 59035 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:2.0:1788571848.596730:0:59073:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f800/0x1e260726fbbbfb32 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbb79 expref: -99 pid: 59035 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000080:00200000:2.0:1788571848.596736:0:59073:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788571848.596740:0:59073:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788571848.596743:0:59073:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbbfb32 00000080:00010000:2.0:1788571848.596746:0:59073:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfb32 from it ffffb68a41f77bb0 00010000:00010000:2.0:1788571848.596749:0:59073:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f800/0x1e260726fbbbfb32 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dbb79 expref: -99 pid: 59035 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:2.0:1788571848.596758:0:59073:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f800/0x1e260726fbbbfb32 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbb79 expref: -99 pid: 59035 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:2.0:1788571848.596766:0:59073:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 44 00000080:00200008:2.0:1788571848.596773:0:59073:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 100000 00000080:00200080:2.0:1788571848.596855:0:59073:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffe2a6c0b0 00800000:00000080:2.0:1788571848.596869:0:59073:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d404929800 uarg=0000000000000000 00800000:00000004:2.0:1788571848.596875:0:59073:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0000-mdc-ffff94d405cc3800: age 2986, max_age 3001 00000100:00100000:2.0:1788571848.596915:0:59073:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:59073:1875450488286336:192.168.202.125@tcp:41:lfs.0 00000100:00100000:2.0:1788571848.596995:0:59073:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4128ff180 going to sleep for 11 seconds 00000100:00100000:2.0:1788571848.691286:0:59083:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:59083:1875450488287104:192.168.202.125@tcp:101:rm.0 00010000:00010000:2.0:1788571848.691311:0:59083:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd000/0x1e260726fbbbfbe1 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbe20 expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571848.691323:0:59083:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd000/0x1e260726fbbbfbe1 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbe20 expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571848.691335:0:59083:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd000/0x1e260726fbbbfbe1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dbe20 expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571848.691346:0:59083:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd000/0x1e260726fbbbfbe1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbe20 expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571848.691358:0:59083:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd000/0x1e260726fbbbfbe1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dbe20 expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571848.691370:0:59083:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd000/0x1e260726fbbbfbe1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dbe20 expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571848.691377:0:59083:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd000/0x1e260726fbbbfbe1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dbe20 expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788571848.691387:0:59083:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408c57b80 x1875450488287104/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/688 e 0 to 0 dl 1788571864 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788571848.691401:0:59083:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd000/0x1e260726fbbbfbe1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbe20 expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571848.691414:0:59083:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788571848.691421:0:59083:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788571848.691451:0:59083:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788571848.691456:0:59083:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbbfbe1 00000080:00010000:2.0:1788571848.691471:0:59083:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788571848.691474:0:59083:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfbe1 from it ffffb68a41f97b58 00010000:00010000:2.0:1788571848.691478:0:59083:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd000/0x1e260726fbbbfbe1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dbe20 expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571848.691487:0:59083:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd000/0x1e260726fbbbfbe1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbe20 expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788571848.691512:0:59083:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788571848.691530:0:59083:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg225-client.virtnet, dir=[0x200000007:0x1:0x0](ffff94d403679988), flags=8 00000080:00200000:2.0:1788571848.691535:0:59083:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=fsa-oleg225-client.virtnet, dir=[0x200000007:0x1:0x0](ffff94d403679988), intent=getattr 00000002:00010000:2.0:1788571848.691558:0:59083:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: fsa-oleg225-client.virtnet,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788571848.691597:0:59083:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbe8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 59083 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571848.691605:0:59083:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbe8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 59083 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571848.691613:0:59083:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbe8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 59083 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788571848.691636:0:59083:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:59083:1875450488287232:192.168.202.125@tcp:101:rm.0 00000100:00100000:2.0:1788571848.691708:0:59083:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4128ffcc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788571848.698381:0:59083:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:59083:1875450488287232:192.168.202.125@tcp:101:rm.0 00010000:00010000:2.0:1788571848.698407:0:59083:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbe8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbe3c expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571848.698431:0:59083:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbe8 lrc: 4/1,0 mode: --/PR res: [0x2000061c1:0x1b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbe3c expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571848.698438:0:59083:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbe8 lrc: 4/1,0 mode: --/PR res: [0x2000061c1:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbe3c expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571848.698449:0:59083:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbe8 lrc: 4/1,0 mode: PR/PR res: [0x2000061c1:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dbe3c expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571848.698471:0:59083:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbe8 lrc: 4/1,0 mode: PR/PR res: [0x2000061c1:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbe3c expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571848.698485:0:59083:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbe8 lrc: 5/2,0 mode: PR/PR res: [0x2000061c1:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dbe3c expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571848.698494:0:59083:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbe8 lrc: 5/2,0 mode: PR/PR res: [0x2000061c1:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dbe3c expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571848.698502:0:59083:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbe8 lrc: 4/1,0 mode: PR/PR res: [0x2000061c1:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dbe3c expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788571848.698514:0:59083:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408c62a00 x1875450488287232/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 600/584 e 0 to 0 dl 1788571864 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788571848.698527:0:59083:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbe8 lrc: 3/1,0 mode: PR/PR res: [0x2000061c1:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbe3c expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571848.698536:0:59083:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589959617/27 (0 0) 00000080:00200000:2.0:1788571848.698543:0:59083:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788571848.698564:0:59083:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x2000061c1:0x1b:0x0], updating i_size 0 00000080:00200000:2.0:1788571848.698567:0:59083:0:(namei.c:147:ll_iget()) got inode: [0x2000061c1:0x1b:0x0](ffff94d4109e5b88): rc = 0 00000080:00200000:2.0:1788571848.698614:0:59083:0:(file.c:6826:ll_layout_conf()) [0x2000061c1:0x1b:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1788571848.698634:0:59083:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000061c1:0x1b:0x0](ffff94d4109e5b88) for lock 0x1e260726fbbbfbe8 00000080:00010000:2.0:1788571848.698643:0:59083:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000061c1:0x1b:0x0](ffff94d4109e5b88) 00000080:00010000:2.0:1788571848.698645:0:59083:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfbe8 from it ffffb68a41f97b20 00010000:00010000:2.0:1788571848.698648:0:59083:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbe8 lrc: 3/1,0 mode: PR/PR res: [0x2000061c1:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4dbe3c expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571848.698654:0:59083:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbe8 lrc: 2/0,0 mode: PR/PR res: [0x2000061c1:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4dbe3c expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788571848.698685:0:59083:0:(file.c:6285:ll_getattr_dentry()) START file fsa-oleg225-client.virtnet:[0x2000061c1:0x1b:0x0](ffff94d4109e5b88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788571848.698691:0:59083:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x2000061c1:0x1b:0x0](ffff94d4109e5b88),name=fsa-oleg225-client.virtnet 00000002:00010000:2.0:1788571848.698709:0:59083:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000061c1:0x1b:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788571848.698715:0:59083:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbe8 lrc: 2/1,0 mode: PR/PR res: [0x2000061c1:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4dbe3c expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788571848.698722:0:59083:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbe8 lrc: 2/1,0 mode: PR/PR res: [0x2000061c1:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4dbe3c expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788571848.698726:0:59083:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788571848.698729:0:59083:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000061c1:0x1b:0x0](ffff94d4109e5b88) 00000080:00010000:2.0:1788571848.698731:0:59083:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000061c1:0x1b:0x0](ffff94d4109e5b88) for lock 0x1e260726fbbbfbe8 00000080:00010000:2.0:1788571848.698733:0:59083:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfbe8 from it ffffb68a41f97cb8 00010000:00010000:2.0:1788571848.698735:0:59083:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbe8 lrc: 3/1,0 mode: PR/PR res: [0x2000061c1:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4dbe3c expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788571848.698740:0:59083:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbe8 lrc: 2/0,0 mode: PR/PR res: [0x2000061c1:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4dbe3c expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788571848.698785:0:59083:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788571848.698789:0:59083:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788571848.698793:0:59083:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd000/0x1e260726fbbbfbe1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbe20 expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788571848.698797:0:59083:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd000/0x1e260726fbbbfbe1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbe20 expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788571848.698801:0:59083:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788571848.698803:0:59083:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788571848.698805:0:59083:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbbfbe1 00000080:00010000:2.0:1788571848.698807:0:59083:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfbe1 from it ffffb68a41f97c10 00010000:00010000:2.0:1788571848.698809:0:59083:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd000/0x1e260726fbbbfbe1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dbe20 expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788571848.698813:0:59083:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd000/0x1e260726fbbbfbe1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbe20 expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788571848.698817:0:59083:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788571848.698823:0:59083:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=fsa-oleg225-client.virtnet, flags=0 00000080:00200000:2.0:1788571848.698826:0:59083:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788571848.698829:0:59083:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788571848.698832:0:59083:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd000/0x1e260726fbbbfbe1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbe20 expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788571848.698836:0:59083:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd000/0x1e260726fbbbfbe1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbe20 expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788571848.698839:0:59083:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788571848.698841:0:59083:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788571848.698843:0:59083:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbbfbe1 00000080:00010000:2.0:1788571848.698844:0:59083:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfbe1 from it ffffb68a41f97d68 00010000:00010000:2.0:1788571848.698846:0:59083:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd000/0x1e260726fbbbfbe1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dbe20 expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788571848.698852:0:59083:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd000/0x1e260726fbbbfbe1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbe20 expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788571848.698865:0:59083:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 3 00000080:00200000:2.0:1788571848.698870:0:59083:0:(namei.c:2418:ll_unlink()) VFS Op:name=fsa-oleg225-client.virtnet, dir=[0x200000007:0x1:0x0](ffff94d403679988) 00010000:00010000:2.0:1788571848.698904:0:59083:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd000/0x1e260726fbbbfbe1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dbe20 expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788571848.698923:0:59083:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbe8 lrc: 2/0,0 mode: PR/PR res: [0x2000061c1:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xec901abfeb4dbe3c expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788571848.698933:0:59083:0:(vvp_object.c:117:vvp_conf_set()) [0x2000061c1:0x1b:0x0]: losing layout lock 00010000:00010000:2.0:1788571848.698963:0:59083:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbe8 lrc: 1/0,0 mode: --/PR res: [0x2000061c1:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xec901abfeb4dbe3c expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788571848.698971:0:59083:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd000/0x1e260726fbbbfbe1 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dbe20 expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788571848.698978:0:59083:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788571848.698980:0:59083:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfbe8 lrc: 0/0,0 mode: --/PR res: [0x2000061c1:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xec901abfeb4dbe3c expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788571848.698995:0:59083:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd000/0x1e260726fbbbfbe1 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dbe20 expref: -99 pid: 59083 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1788571848.699012:0:59083:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788571848.699017:0:59083:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c62a00 x1875450488287488/t0(0) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 512/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788571848.699035:0:59083:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:59083:1875450488287488:192.168.202.125@tcp:36:rm.0 00000100:00100000:2.0:1788571848.699098:0:59083:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4128ffcc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788571848.720715:0:59083:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:59083:1875450488287488:192.168.202.125@tcp:36:rm.0 00000080:00200000:2.0:1788571848.720812:0:59083:0:(vvp_io.c:1826:vvp_io_init()) [0x2000061c1:0x1b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788571848.720821:0:59083:0:(vvp_io.c:304:vvp_io_fini()) [0x2000061c1:0x1b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788571848.720981:0:59083:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c1:0x1b:0x0](ffff94d4109e5b88) 00000100:00100000:2.0:1788571855.867325:0:2396:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.125@tcp (source: 12345-192.168.202.125@tcp) 00010000:00010000:2.0:1788571855.946752:0:59151:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 30 res 8589959617/23 (0 0) 00000080:00200000:2.0:1788571855.946825:0:59151:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788571855.946839:0:59151:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788571855.946848:0:59151:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a3400/0x1e260726fbbbfbef lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbe66 expref: -99 pid: 59150 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788571855.946859:0:59151:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a3400/0x1e260726fbbbfbef lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbe66 expref: -99 pid: 59150 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1788571855.946866:0:59151:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788571855.946871:0:59151:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788571855.946874:0:59151:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbbfbef 00000080:00010000:2.0:1788571855.946879:0:59151:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfbef from it ffffb68a420b7be8 00010000:00010000:2.0:1788571855.946883:0:59151:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a3400/0x1e260726fbbbfbef lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dbe66 expref: -99 pid: 59150 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788571855.946892:0:59151:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a3400/0x1e260726fbbbfbef lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dbe66 expref: -99 pid: 59150 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1788571855.946905:0:59151:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788571855.946912:0:59151:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d118.replay-single, flags=16408 00000080:00200000:2.0:1788571855.946917:0:59151:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x2000061c1:0x17:0x0](ffff94d4102242c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788571855.946928:0:59151:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=striped_dir1, dir=[0x2000061c1:0x17:0x0](ffff94d4102242c8), flags=16392 00000080:00200000:2.0:1788571855.946932:0:59151:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=striped_dir1, dir=[0x2000061c1:0x17:0x0](ffff94d4102242c8), intent=getattr 00000002:00010000:2.0:1788571855.946941:0:59151:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: striped_dir1,[0x0:0x0:0x0]) in obj [0x2000061c1:0x17:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788571855.946975:0:59151:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f000/0x1e260726fbbbfc12 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x17:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 59151 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571855.946982:0:59151:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f000/0x1e260726fbbbfc12 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x17:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 59151 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571855.946992:0:59151:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f000/0x1e260726fbbbfc12 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x17:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 59151 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788571855.947014:0:59151:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:59151:1875450488289920:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788571855.947194:0:59151:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4128ff840 going to sleep for 11 seconds 00010000:00010000:2.0:1788571859.591859:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00100000:2.0:1788571860.727363:0:2406:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788571860.727404:0:2406:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62300 x1875450488291072/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788571860.727441:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488290560:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788571860.727494:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488290688:192.168.202.125@tcp:400:kworker.0 00010000:00010000:2.0:1788571861.638928:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788571863.685599:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788571865.734707:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788571865.803770:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788571865.803777:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788571865.803781:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788571865.803792:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3014 00000100:00080000:2.0:1788571865.803803:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: tried all connections, increasing latency to 5s 00000100:00080000:2.0:1788571865.803817:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788571865.803828:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788571865.803848:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488291328/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788571865.803919:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488291456:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788571865.804124:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488291712:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788571865.804173:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:2.0:1788571865.804178:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488291200:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788571865.804267:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488291584:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788571865.811053:0:2406:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff94d405cc3800: committing for last_committed 81604378634 gen 1 00000100:00100000:2.0:1788571865.811165:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488291456:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788571865.811185:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488291712:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788571865.822453:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488291584:192.168.202.125@tcp:400:kworker.0 00010000:00010000:2.0:1788571867.781465:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788571869.829726:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788571870.936891:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488292096:192.168.202.125@tcp:400:kworker.0 00010000:00010000:2.0:1788571871.877600:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788571873.926989:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788571875.973639:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1788571876.037375:0:2406:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788571860/real 1788571860] req@ffff94d408c60700 x1875450488290432/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1788571876 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788571876.100804:0:2406:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff94d408c60700 x1875450488290432/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1788571876 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788571876.100823:0:2406:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from FULL to DISCONN 00000100:02020000:2.0:1788571876.100836:0:2406:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.202.125@tcp: Connection to MGS (at 192.168.202.125@tcp) was lost; in progress operations using this service will fail 00000100:00080000:2.0:1788571876.118639:0:2406:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.202.125@tcp for MGC192.168.202.125@tcp not replayable, auto-deactivating 00000100:00080000:2.0:1788571876.118644:0:2406:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:2.0:1788571876.118653:0:2406:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff94d408c60700 x1875450488290432/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1788571876 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788571876.118667:0:2406:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff94d408c62680 x1875450488291200/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 0 dl 1788571881 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788571876.118676:0:2406:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff94d408c57480 x1875450488291840/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 0 dl 1788571886 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788571876.118685:0:2406:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff94d412926d80 x1875450488292480/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 0 dl 1788571892 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:2.0:1788571876.118759:0:2406:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:2.0:1788571876.118763:0:2406:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:2.0:1788571876.118765:0:2406:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:2.0:1788571876.118767:0:2406:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00010000:00010000:2.0:1788571878.021815:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788571880.069542:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788571882.120385:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788571884.171528:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788571886.218337:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788571886.286571:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d412927800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e60431de-58b9-4f69-9048-11a3d82ddcdd:2402:1875450488293504:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788571887.598866:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4125b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488293888:192.168.202.125@tcp:38:ll_cfg_requeue.0 00000040:00080000:2.0:1788571887.668937:0:59385:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 64768 count 51 00000040:00080000:2.0:1788571887.713371:0:59386:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 64768 count 7 00000100:00100000:2.0:1788571892.422901:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4125b1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488290304:192.168.202.125@tcp:36:lfs.0 00000100:00100000:2.0:1788571892.424297:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4125b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488295296:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00100000:2.0:1788571892.424380:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4125b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488295424:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00100000:2.0:1788571892.424402:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4125b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488295552:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000080:00200080:2.0:1788571896.879930:0:59476:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff708081b0 00020000:00000080:2.0:1788571896.879953:0:59476:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d404928c00 uarg=0000000000000000 00020000:00000004:2.0:1788571896.879961:0:59476:0:(obd_class.h:1091:obd_statfs()) lustre-OST0001-osc-ffff94d405cc3800: age 3002, max_age 3049 00000100:00080000:2.0:1788571896.879974:0:59476:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 2975 00000100:00080000:2.0:1788571896.879988:0:59476:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d405cc3800: import ffff94d405bab800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788571896.879998:0:59476:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8437760, lost: 0. 00000100:00100000:2.0:1788571896.880025:0:59476:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488296704/t0(0) o8->lustre-OST0001-osc-ffff94d405cc3800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788571896.880064:0:59476:0:(client.c:1832:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff94d408c62d80 x1875450488296832/t0(0) o13->lustre-OST0001-osc-ffff94d405cc3800@192.168.202.125@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788571896.880075:0:59476:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4128ffa80 going to sleep for 0 seconds 00000100:00100000:2.0:1788571896.888654:0:59476:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4128ffa80 going to sleep for 11 seconds 00000100:00100000:2.0:1788571898.597162:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488297088:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788571898.597180:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488297344:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788571898.597190:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488296960:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788571898.597200:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488297216:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788571898.597226:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488297472:192.168.202.125@tcp:400:kworker.0 00010000:00010000:2.0:1788571907.719894:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788571908.808634:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488298880:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788571908.808653:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488299136:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788571908.808662:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488298752:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788571908.808671:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488299008:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788571908.808681:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488299264:192.168.202.125@tcp:400:kworker.0 00000080:00200000:2.0:1788571915.597748:0:60561:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788571915.597778:0:60561:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788571915.597836:0:60561:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e400/0x1e260726fbbbfc43 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 60561 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571915.597846:0:60561:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e400/0x1e260726fbbbfc43 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 60561 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571915.597856:0:60561:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e400/0x1e260726fbbbfc43 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 60561 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788571915.599193:0:60561:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job mkdir:9bf287c0-f582-4b83-84c1-15e766218620:60561:1875450488300416:192.168.202.125@tcp:101:mkdir.0 00000100:00100000:2.0:1788571915.695632:0:60562:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4125b0000 pname:cluuid:pid:xid:nid:opc:job rmdir:9bf287c0-f582-4b83-84c1-15e766218620:60562:1875450488300928:192.168.202.125@tcp:101:rmdir.0 00010000:00010000:2.0:1788571915.695660:0:60562:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a3000/0x1e260726fbbbfc51 lrc: 4/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc40f expref: -99 pid: 60562 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571915.695686:0:60562:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a3000/0x1e260726fbbbfc51 lrc: 4/1,0 mode: --/PR res: [0x2000061c1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc40f expref: -99 pid: 60562 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571915.695695:0:60562:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a3000/0x1e260726fbbbfc51 lrc: 4/1,0 mode: --/PR res: [0x2000061c1:0x1e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc40f expref: -99 pid: 60562 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571915.695716:0:60562:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a3000/0x1e260726fbbbfc51 lrc: 4/1,0 mode: PR/PR res: [0x2000061c1:0x1e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dc40f expref: -99 pid: 60562 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571915.695727:0:60562:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a3000/0x1e260726fbbbfc51 lrc: 4/1,0 mode: PR/PR res: [0x2000061c1:0x1e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc40f expref: -99 pid: 60562 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571915.695742:0:60562:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a3000/0x1e260726fbbbfc51 lrc: 5/2,0 mode: PR/PR res: [0x2000061c1:0x1e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dc40f expref: -99 pid: 60562 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571915.695751:0:60562:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a3000/0x1e260726fbbbfc51 lrc: 5/2,0 mode: PR/PR res: [0x2000061c1:0x1e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dc40f expref: -99 pid: 60562 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571915.695759:0:60562:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a3000/0x1e260726fbbbfc51 lrc: 4/1,0 mode: PR/PR res: [0x2000061c1:0x1e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dc40f expref: -99 pid: 60562 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788571915.695771:0:60562:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d4125b0000 x1875450488300928/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/584 e 0 to 0 dl 1788571931 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788571915.695785:0:60562:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a3000/0x1e260726fbbbfc51 lrc: 3/1,0 mode: PR/PR res: [0x2000061c1:0x1e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc40f expref: -99 pid: 60562 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571915.695796:0:60562:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589959617/30 (0 0) 00000080:00200000:2.0:1788571915.695804:0:60562:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2000061c1:0x1d:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788571915.695829:0:60562:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x2000061c1:0x1e:0x0], updating i_size 4096 00000080:00200000:2.0:1788571915.695832:0:60562:0:(namei.c:147:ll_iget()) got inode: [0x2000061c1:0x1e:0x0](ffff94d410bd00c8): rc = 0 00000080:00010000:2.0:1788571915.695838:0:60562:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000061c1:0x1e:0x0](ffff94d410bd00c8) for lock 0x1e260726fbbbfc51 00000080:00010000:2.0:1788571915.695855:0:60562:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000061c1:0x1e:0x0](ffff94d410bd00c8) 00000080:00010000:2.0:1788571915.695860:0:60562:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfc51 from it ffffb68a41067e08 00010000:00010000:2.0:1788571915.695863:0:60562:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a3000/0x1e260726fbbbfc51 lrc: 3/1,0 mode: PR/PR res: [0x2000061c1:0x1e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dc40f expref: -99 pid: 60562 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571915.695871:0:60562:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a3000/0x1e260726fbbbfc51 lrc: 2/0,0 mode: PR/PR res: [0x2000061c1:0x1e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc40f expref: -99 pid: 60562 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788571915.695904:0:60562:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), inode mode 41ed mask 3 00000080:00200000:2.0:1788571915.695912:0:60562:0:(namei.c:2328:ll_rmdir()) VFS Op:name=tmp, dir=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88) 00010000:00010000:2.0:1788571915.695942:0:60562:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a3000/0x1e260726fbbbfc51 lrc: 2/0,0 mode: PR/PR res: [0x2000061c1:0x1e:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dc40f expref: -99 pid: 60562 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788571915.695990:0:60562:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a3000/0x1e260726fbbbfc51 lrc: 1/0,0 mode: --/PR res: [0x2000061c1:0x1e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dc40f expref: -99 pid: 60562 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788571915.695997:0:60562:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788571915.696001:0:60562:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a3000/0x1e260726fbbbfc51 lrc: 0/0,0 mode: --/PR res: [0x2000061c1:0x1e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dc40f expref: -99 pid: 60562 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788571915.696024:0:60562:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788571915.696031:0:60562:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c61500 x1875450488301184/t0(0) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788571915.696054:0:60562:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job rmdir:9bf287c0-f582-4b83-84c1-15e766218620:60562:1875450488301184:192.168.202.125@tcp:36:rmdir.0 00000100:00100000:2.0:1788571915.696146:0:60562:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d41187bc00 going to sleep for 11 seconds 00000001:02000400:2.0:1788571925.140501:0:60647:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 00000100:00100000:2.0:1788571925.300872:0:60651:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:60651:1875450488305408:192.168.202.125@tcp:101:lfs.0 00010000:00010000:2.0:1788571925.300893:0:60651:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddcc00/0x1e260726fbbbfc89 lrc: 4/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 60651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.300906:0:60651:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddcc00/0x1e260726fbbbfc89 lrc: 4/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 60651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.300915:0:60651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddcc00/0x1e260726fbbbfc89 lrc: 4/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 60651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.300922:0:60651:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddcc00/0x1e260726fbbbfc89 lrc: 3/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 60651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.300931:0:60651:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddcc00/0x1e260726fbbbfc89 lrc: 4/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 60651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.300937:0:60651:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d408ddcc00 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:2.0:1788571925.300941:0:60651:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddcc00/0x1e260726fbbbfc89 lrc: 4/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 60651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.300949:0:60651:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddcc00/0x1e260726fbbbfc89 lrc: 4/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 60651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.300956:0:60651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddcc00/0x1e260726fbbbfc89 lrc: 5/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 60651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.300962:0:60651:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788571925.300968:0:60651:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddcc00/0x1e260726fbbbfc89 lrc: 3/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 60651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.300975:0:60651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddcc00/0x1e260726fbbbfc89 lrc: 0/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 60651 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788571925.300992:0:60651:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff94d408c54000 x1875450488305408/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 584/584 e 0 to 0 dl 1788571941 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788571925.301009:0:60651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2000061c1:0x1d:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1788571925.301078:0:60651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 30 res 8589959617/29 (0 0) 00000080:00200000:2.0:1788571925.301156:0:60651:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788571925.301171:0:60651:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788571925.301189:0:60651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ee00/0x1e260726fbbbfc66 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc486 expref: -99 pid: 60648 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1788571925.301610:0:60651:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ee00/0x1e260726fbbbfc66 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc486 expref: -99 pid: 60648 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1788571925.301620:0:60651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788571925.301627:0:60651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788571925.301630:0:60651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbbfc66 00000080:00010000:2.0:1788571925.301635:0:60651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfc66 from it ffffb68a4205fbe8 00010000:00010000:2.0:1788571925.301639:0:60651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ee00/0x1e260726fbbbfc66 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dc486 expref: -99 pid: 60648 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1788571925.301647:0:60651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ee00/0x1e260726fbbbfc66 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc486 expref: -99 pid: 60648 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1788571925.301662:0:60651:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788571925.301669:0:60651:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d119.replay-single, flags=16408 00000080:00200000:2.0:1788571925.301675:0:60651:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788571925.301686:0:60651:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=stripe_dir-1, dir=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), flags=16392 00000080:00200000:2.0:1788571925.301691:0:60651:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-1, dir=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), intent=getattr 00000002:00010000:2.0:1788571925.301703:0:60651:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: stripe_dir-1,[0x0:0x0:0x0]) in obj [0x2000061c1:0x1d:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788571925.301745:0:60651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfc90 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 60651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.301753:0:60651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfc90 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 60651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.301762:0:60651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfc90 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 60651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788571925.301781:0:60651:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:60651:1875450488305536:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788571925.301856:0:60651:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408619cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788571925.307069:0:60651:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:60651:1875450488305536:192.168.202.125@tcp:101:lfs.0 00010000:00010000:2.0:1788571925.307092:0:60651:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfc90 lrc: 4/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 60651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.307117:0:60651:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfc90 lrc: 4/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 60651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.307124:0:60651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfc90 lrc: 4/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 60651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.307130:0:60651:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfc90 lrc: 3/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 60651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.307139:0:60651:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfc90 lrc: 4/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 60651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.307150:0:60651:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d41251f400 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:2.0:1788571925.307154:0:60651:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfc90 lrc: 4/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 60651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.307161:0:60651:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfc90 lrc: 4/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 60651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.307168:0:60651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfc90 lrc: 5/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 60651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.307175:0:60651:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788571925.307190:0:60651:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfc90 lrc: 3/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 60651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.307197:0:60651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbbfc90 lrc: 0/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 60651 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788571925.307246:0:60651:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff94d408c62300 x1875450488305536/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 584/584 e 0 to 0 dl 1788571941 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788571925.307262:0:60651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2000061c1:0x1d:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1788571925.307288:0:60651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 30 res 8589959617/29 (0 0) 00000080:00200000:2.0:1788571925.307363:0:60651:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788571925.307377:0:60651:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788571925.307396:0:60651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ee00/0x1e260726fbbbfc66 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc486 expref: -99 pid: 60648 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1788571925.307407:0:60651:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ee00/0x1e260726fbbbfc66 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc486 expref: -99 pid: 60648 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1788571925.307418:0:60651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788571925.307423:0:60651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788571925.307426:0:60651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbbfc66 00000080:00010000:2.0:1788571925.307431:0:60651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfc66 from it ffffb68a4205fb58 00010000:00010000:2.0:1788571925.307434:0:60651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ee00/0x1e260726fbbbfc66 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dc486 expref: -99 pid: 60648 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1788571925.307442:0:60651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ee00/0x1e260726fbbbfc66 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc486 expref: -99 pid: 60648 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:2.0:1788571925.307455:0:60651:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788571925.307462:0:60651:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d119.replay-single, flags=13 00000080:00200008:2.0:1788571925.307471:0:60651:0:(file.c:6285:ll_getattr_dentry()) START file d119.replay-single:[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788571925.307476:0:60651:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88),name=d119.replay-single 00000002:00010000:2.0:1788571925.307488:0:60651:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000061c1:0x1d:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788571925.307492:0:60651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 30 res 8589959617/29 (0 0) 00010000:00010000:2.0:1788571925.307541:0:60651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f600/0x1e260726fbbbfc97 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 60651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.307549:0:60651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f600/0x1e260726fbbbfc97 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 60651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.307559:0:60651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f600/0x1e260726fbbbfc97 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 60651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788571925.307579:0:60651:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:60651:1875450488305664:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788571925.307651:0:60651:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408619cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788571925.313053:0:60651:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:60651:1875450488305664:192.168.202.125@tcp:101:lfs.0 00010000:00010000:2.0:1788571925.313082:0:60651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f600/0x1e260726fbbbfc97 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc543 expref: -99 pid: 60651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.313110:0:60651:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f600/0x1e260726fbbbfc97 lrc: 4/1,0 mode: --/PR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc543 expref: -99 pid: 60651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.313120:0:60651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f600/0x1e260726fbbbfc97 lrc: 4/1,0 mode: --/PR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc543 expref: -99 pid: 60651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.313132:0:60651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f600/0x1e260726fbbbfc97 lrc: 4/1,0 mode: PR/PR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dc543 expref: -99 pid: 60651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.313142:0:60651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f600/0x1e260726fbbbfc97 lrc: 4/1,0 mode: PR/PR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc543 expref: -99 pid: 60651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.313155:0:60651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f600/0x1e260726fbbbfc97 lrc: 5/2,0 mode: PR/PR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dc543 expref: -99 pid: 60651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.313163:0:60651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f600/0x1e260726fbbbfc97 lrc: 5/2,0 mode: PR/PR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dc543 expref: -99 pid: 60651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.313170:0:60651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f600/0x1e260726fbbbfc97 lrc: 4/1,0 mode: PR/PR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dc543 expref: -99 pid: 60651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788571925.313194:0:60651:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408c62300 x1875450488305664/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/688 e 0 to 0 dl 1788571941 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788571925.313244:0:60651:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f600/0x1e260726fbbbfc97 lrc: 3/1,0 mode: PR/PR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc543 expref: -99 pid: 60651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.313255:0:60651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589959617/29 (0 0) 00000080:00200000:2.0:1788571925.313263:0:60651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788571925.313291:0:60651:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x2000061c1:0x1d:0x0], updating i_size 4096 00000080:00010000:2.0:1788571925.313297:0:60651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000061c1:0x1d:0x0](ffff94d410bd3a88) for lock 0x1e260726fbbbfc97 00000080:00010000:2.0:1788571925.313305:0:60651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000061c1:0x1d:0x0](ffff94d410bd3a88) 00000080:00010000:2.0:1788571925.313308:0:60651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfc97 from it ffffb68a4205fcb8 00010000:00010000:2.0:1788571925.313311:0:60651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f600/0x1e260726fbbbfc97 lrc: 3/1,0 mode: PR/PR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dc543 expref: -99 pid: 60651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.313318:0:60651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f600/0x1e260726fbbbfc97 lrc: 2/0,0 mode: PR/PR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc543 expref: -99 pid: 60651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788571925.352933:0:60652:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:60652:1875450488306304:192.168.202.125@tcp:101:lfs.0 00010000:00010000:2.0:1788571925.352952:0:60652:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd600/0x1e260726fbbbfca5 lrc: 4/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 60652 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.352964:0:60652:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd600/0x1e260726fbbbfca5 lrc: 4/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 60652 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.352969:0:60652:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd600/0x1e260726fbbbfca5 lrc: 4/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 60652 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.352974:0:60652:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd600/0x1e260726fbbbfca5 lrc: 3/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 60652 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.352980:0:60652:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd600/0x1e260726fbbbfca5 lrc: 4/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 60652 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.352987:0:60652:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d408ddd600 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:2.0:1788571925.352991:0:60652:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd600/0x1e260726fbbbfca5 lrc: 4/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 60652 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.352996:0:60652:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd600/0x1e260726fbbbfca5 lrc: 4/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 60652 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.353000:0:60652:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd600/0x1e260726fbbbfca5 lrc: 5/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x20e400000000 pid: 60652 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.353005:0:60652:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788571925.353009:0:60652:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd600/0x1e260726fbbbfca5 lrc: 3/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 60652 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.353013:0:60652:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd600/0x1e260726fbbbfca5 lrc: 0/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 60652 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788571925.353030:0:60652:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff94d408c57100 x1875450488306304/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 584/584 e 0 to 0 dl 1788571941 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788571925.353044:0:60652:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2000061c1:0x1d:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1788571925.353073:0:60652:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 30 res 8589959617/29 (0 0) 00000080:00200000:2.0:1788571925.353183:0:60652:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788571925.353199:0:60652:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788571925.353447:0:60652:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ee00/0x1e260726fbbbfc66 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc486 expref: -99 pid: 60648 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:2.0:1788571925.353460:0:60652:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ee00/0x1e260726fbbbfc66 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc486 expref: -99 pid: 60648 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:2.0:1788571925.353470:0:60652:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788571925.353477:0:60652:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788571925.353480:0:60652:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbbfc66 00000080:00010000:2.0:1788571925.353484:0:60652:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfc66 from it ffffb68a4206fbe8 00010000:00010000:2.0:1788571925.353487:0:60652:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ee00/0x1e260726fbbbfc66 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dc486 expref: -99 pid: 60648 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:2.0:1788571925.353493:0:60652:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ee00/0x1e260726fbbbfc66 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc486 expref: -99 pid: 60648 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:2.0:1788571925.353505:0:60652:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788571925.353510:0:60652:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d119.replay-single, flags=16408 00000080:00200000:2.0:1788571925.353514:0:60652:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788571925.353540:0:60652:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=stripe_dir-2, dir=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), flags=16392 00000080:00200000:2.0:1788571925.353548:0:60652:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-2, dir=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), intent=getattr 00000002:00010000:2.0:1788571925.353562:0:60652:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: stripe_dir-2,[0x0:0x0:0x0]) in obj [0x2000061c1:0x1d:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788571925.353604:0:60652:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbbfcac lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 60652 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.353611:0:60652:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbbfcac lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 60652 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.353622:0:60652:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbbfcac lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 60652 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788571925.353645:0:60652:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:60652:1875450488306432:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788571925.353759:0:60652:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408619cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788571925.355770:0:60652:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:60652:1875450488306432:192.168.202.125@tcp:101:lfs.0 00010000:00010000:2.0:1788571925.355786:0:60652:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbbfcac lrc: 4/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 60652 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.355795:0:60652:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbbfcac lrc: 4/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 60652 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.355800:0:60652:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbbfcac lrc: 4/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 60652 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.355804:0:60652:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbbfcac lrc: 3/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 60652 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.355809:0:60652:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbbfcac lrc: 4/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 60652 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.355812:0:60652:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d41251f200 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:2.0:1788571925.355815:0:60652:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbbfcac lrc: 4/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 60652 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.355819:0:60652:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbbfcac lrc: 4/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 60652 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.355823:0:60652:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbbfcac lrc: 5/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x20e400000000 pid: 60652 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.355827:0:60652:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788571925.355831:0:60652:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbbfcac lrc: 3/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 60652 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.355835:0:60652:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbbfcac lrc: 0/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 60652 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788571925.355849:0:60652:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff94d408c62300 x1875450488306432/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 584/584 e 0 to 0 dl 1788571941 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788571925.355860:0:60652:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2000061c1:0x1d:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1788571925.355879:0:60652:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 30 res 8589959617/29 (0 0) 00000080:00200000:2.0:1788571925.355940:0:60652:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788571925.355950:0:60652:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788571925.355956:0:60652:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ee00/0x1e260726fbbbfc66 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc486 expref: -99 pid: 60648 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:2.0:1788571925.355963:0:60652:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ee00/0x1e260726fbbbfc66 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc486 expref: -99 pid: 60648 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:2.0:1788571925.355968:0:60652:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788571925.355972:0:60652:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788571925.355975:0:60652:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbbfc66 00000080:00010000:2.0:1788571925.355978:0:60652:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfc66 from it ffffb68a4206fb58 00010000:00010000:2.0:1788571925.355980:0:60652:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ee00/0x1e260726fbbbfc66 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dc486 expref: -99 pid: 60648 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:2.0:1788571925.355985:0:60652:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ee00/0x1e260726fbbbfc66 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc486 expref: -99 pid: 60648 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:2.0:1788571925.355993:0:60652:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788571925.355998:0:60652:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d119.replay-single, flags=13 00000080:00200008:2.0:1788571925.356004:0:60652:0:(file.c:6285:ll_getattr_dentry()) START file d119.replay-single:[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788571925.356008:0:60652:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88),name=d119.replay-single 00000002:00010000:2.0:1788571925.356016:0:60652:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000061c1:0x1d:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788571925.356020:0:60652:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 30 res 8589959617/29 (0 0) 00010000:00010000:2.0:1788571925.356051:0:60652:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fe00/0x1e260726fbbbfcb3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 60652 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.356058:0:60652:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fe00/0x1e260726fbbbfcb3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 60652 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.356068:0:60652:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fe00/0x1e260726fbbbfcb3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 60652 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788571925.356085:0:60652:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:60652:1875450488306560:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788571925.356413:0:60652:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408619cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788571925.358163:0:60652:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:60652:1875450488306560:192.168.202.125@tcp:101:lfs.0 00010000:00010000:2.0:1788571925.358195:0:60652:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fe00/0x1e260726fbbbfcb3 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc5ba expref: -99 pid: 60652 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.358243:0:60652:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fe00/0x1e260726fbbbfcb3 lrc: 4/1,0 mode: --/PR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc5ba expref: -99 pid: 60652 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.358249:0:60652:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fe00/0x1e260726fbbbfcb3 lrc: 4/1,0 mode: --/PR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc5ba expref: -99 pid: 60652 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.358258:0:60652:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fe00/0x1e260726fbbbfcb3 lrc: 4/1,0 mode: PR/PR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dc5ba expref: -99 pid: 60652 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.358267:0:60652:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fe00/0x1e260726fbbbfcb3 lrc: 4/1,0 mode: PR/PR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc5ba expref: -99 pid: 60652 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.358284:0:60652:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fe00/0x1e260726fbbbfcb3 lrc: 5/2,0 mode: PR/PR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dc5ba expref: -99 pid: 60652 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.358291:0:60652:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fe00/0x1e260726fbbbfcb3 lrc: 5/2,0 mode: PR/PR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dc5ba expref: -99 pid: 60652 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.358298:0:60652:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fe00/0x1e260726fbbbfcb3 lrc: 4/1,0 mode: PR/PR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dc5ba expref: -99 pid: 60652 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788571925.358310:0:60652:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408c62300 x1875450488306560/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/688 e 0 to 0 dl 1788571941 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788571925.358324:0:60652:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fe00/0x1e260726fbbbfcb3 lrc: 3/1,0 mode: PR/PR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc5ba expref: -99 pid: 60652 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.358331:0:60652:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589959617/29 (0 0) 00000080:00200000:2.0:1788571925.358337:0:60652:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788571925.358361:0:60652:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x2000061c1:0x1d:0x0], updating i_size 4096 00000080:00010000:2.0:1788571925.358365:0:60652:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000061c1:0x1d:0x0](ffff94d410bd3a88) for lock 0x1e260726fbbbfcb3 00000080:00010000:2.0:1788571925.358370:0:60652:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000061c1:0x1d:0x0](ffff94d410bd3a88) 00000080:00010000:2.0:1788571925.358372:0:60652:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfcb3 from it ffffb68a4206fcb8 00010000:00010000:2.0:1788571925.358374:0:60652:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fe00/0x1e260726fbbbfcb3 lrc: 3/1,0 mode: PR/PR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dc5ba expref: -99 pid: 60652 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.358380:0:60652:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fe00/0x1e260726fbbbfcb3 lrc: 2/0,0 mode: PR/PR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc5ba expref: -99 pid: 60652 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788571925.358909:0:60652:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788571925.358914:0:60652:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788571925.358926:0:60652:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788571925.358935:0:60652:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ee00/0x1e260726fbbbfc66 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc486 expref: -99 pid: 60648 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:2.0:1788571925.358943:0:60652:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ee00/0x1e260726fbbbfc66 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc486 expref: -99 pid: 60648 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:2.0:1788571925.358947:0:60652:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788571925.358951:0:60652:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788571925.358953:0:60652:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbbfc66 00000080:00010000:2.0:1788571925.358957:0:60652:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfc66 from it ffffb68a4206fcb8 00010000:00010000:2.0:1788571925.358959:0:60652:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ee00/0x1e260726fbbbfc66 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dc486 expref: -99 pid: 60648 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:2.0:1788571925.358963:0:60652:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ee00/0x1e260726fbbbfc66 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc486 expref: -99 pid: 60648 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:2.0:1788571925.359068:0:60652:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788571925.359077:0:60652:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788571925.359082:0:60652:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ee00/0x1e260726fbbbfc66 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc486 expref: -99 pid: 60648 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:2.0:1788571925.359088:0:60652:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ee00/0x1e260726fbbbfc66 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc486 expref: -99 pid: 60648 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:2.0:1788571925.359096:0:60652:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788571925.359100:0:60652:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788571925.359103:0:60652:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbbfc66 00000080:00010000:2.0:1788571925.359106:0:60652:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfc66 from it ffffb68a4206fbf8 00010000:00010000:2.0:1788571925.359109:0:60652:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ee00/0x1e260726fbbbfc66 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dc486 expref: -99 pid: 60648 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:2.0:1788571925.359117:0:60652:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ee00/0x1e260726fbbbfc66 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc486 expref: -99 pid: 60648 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:2.0:1788571925.359126:0:60652:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788571925.359132:0:60652:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d119.replay-single, flags=267 00000080:00200000:2.0:1788571925.359137:0:60652:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), inode mode 41ed mask 44 00000080:00200008:2.0:1788571925.359142:0:60652:0:(file.c:1090:ll_file_open()) START file d119.replay-single:[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), flags 300000 00000080:00200080:2.0:1788571925.359159:0:60652:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88) cmd=c03066f0 arg=7ffd494edb90 00000080:00200000:2.0:1788571925.359168:0:60652:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88) name=stripe_dir-2 stripe_offset=0 stripe_count=2, hash_type=0 00010000:00010000:2.0:1788571925.359190:0:60652:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fe00/0x1e260726fbbbfcb3 lrc: 2/0,0 mode: PR/PR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dc5ba expref: -99 pid: 60652 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788571925.359243:0:60652:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a3400/0x1e260726fbbbfc4a lrc: 2/0,0 mode: PR/PR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x11/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc3de expref: -99 pid: 60561 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788571925.359284:0:60652:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fe00/0x1e260726fbbbfcb3 lrc: 1/0,0 mode: --/PR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dc5ba expref: -99 pid: 60652 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788571925.359292:0:60652:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788571925.359295:0:60652:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fe00/0x1e260726fbbbfcb3 lrc: 0/0,0 mode: --/PR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dc5ba expref: -99 pid: 60652 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788571925.359312:0:60652:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788571925.359320:0:60652:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c62300 x1875450488306816/t0(0) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788571925.359342:0:60652:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:60652:1875450488306816:192.168.202.125@tcp:36:lfs.0 00000100:00100000:2.0:1788571925.359409:0:60652:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408619cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788571925.365669:0:60652:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:60652:1875450488306816:192.168.202.125@tcp:36:lfs.0 00000080:00200000:2.0:1788571925.365726:0:60652:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x2000061c1:0x23:0x0](ffff94d40367ba88) 00000080:00200000:2.0:1788571925.365734:0:60652:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x2000061c1:0x23:0x0], updating i_size 4096 00000080:00200008:2.0:1788571925.365767:0:60652:0:(file.c:479:ll_file_release()) START file d119.replay-single:[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), flags 300000 00010000:00010000:2.0:1788571925.365796:0:60652:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddde00/0x1e260726fbbbfc9e lrc: 2/0,0 mode: CR/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc551 expref: -99 pid: 60651 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788571925.610962:0:60659:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788571925.610995:0:60659:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788571925.611015:0:60659:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ee00/0x1e260726fbbbfc66 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc486 expref: -99 pid: 60648 timeout: 0 lvb_type: 0 lru_score: 46 lru_type: 1 00010000:00010000:2.0:1788571925.611034:0:60659:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ee00/0x1e260726fbbbfc66 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc486 expref: -99 pid: 60648 timeout: 0 lvb_type: 0 lru_score: 46 lru_type: 1 00000080:00200000:2.0:1788571925.611044:0:60659:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788571925.611051:0:60659:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788571925.611055:0:60659:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbbfc66 00000080:00010000:2.0:1788571925.611063:0:60659:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfc66 from it ffffb68a41fffb58 00010000:00010000:2.0:1788571925.611067:0:60659:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ee00/0x1e260726fbbbfc66 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dc486 expref: -99 pid: 60648 timeout: 0 lvb_type: 0 lru_score: 46 lru_type: 1 00010000:00010000:2.0:1788571925.611076:0:60659:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ee00/0x1e260726fbbbfc66 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc486 expref: -99 pid: 60648 timeout: 0 lvb_type: 0 lru_score: 47 lru_type: 1 00000080:00200000:2.0:1788571925.611093:0:60659:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788571925.611174:0:60659:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d119.replay-single, flags=29 00000080:00200000:2.0:1788571925.611181:0:60659:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788571925.611193:0:60659:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=stripe_dir-9, dir=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), flags=13 00000080:00200000:2.0:1788571925.611199:0:60659:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-9, dir=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), intent=getattr 00000002:00010000:2.0:1788571925.611406:0:60659:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: stripe_dir-9,[0x0:0x0:0x0]) in obj [0x2000061c1:0x1d:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788571925.611453:0:60659:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fa00/0x1e260726fbbbfd38 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 60659 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.611462:0:60659:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fa00/0x1e260726fbbbfd38 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 60659 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571925.611472:0:60659:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fa00/0x1e260726fbbbfd38 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 60659 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788571925.611497:0:60659:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:60659:1875450488310784:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788571925.611760:0:60659:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408619cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788571929.290706:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488318208:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788571934.406228:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4125b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488318720:192.168.202.125@tcp:38:kworker.0 00000100:00080000:2.0:1788571934.431933:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788571934.431959:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-19) 00000100:00100000:2.0:1788571934.431966:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4125b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488318720:192.168.202.125@tcp:38:kworker.0 00000100:00080000:2.0:1788571939.530075:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788571939.530082:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788571939.530086:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788571939.530095:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3087 00000100:00080000:2.0:1788571939.530106:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: tried all connections, increasing latency to 10s 00000100:00080000:2.0:1788571939.530114:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788571939.530125:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788571939.530143:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488319360/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788571939.530191:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405bad000 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00080000:2.0:1788571939.530194:0:51031:0:(import.c:2101:ptlrpc_disconnect_and_idle_import()) lustre-OST0000-osc-ffff94d405cc3800: disconnect after 23s idle 00000100:00080000:2.0:1788571939.530240:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405bab800 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:00080000:2.0:1788571939.530243:0:51031:0:(import.c:2101:ptlrpc_disconnect_and_idle_import()) lustre-OST0001-osc-ffff94d405cc3800: disconnect after 23s idle 00000100:00100000:2.0:1788571939.530396:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488319232:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788571939.530497:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788571939.530505:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488319488:192.168.202.125@tcp:400:kworker.0 00000100:00080000:2.0:1788571944.645521:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788571944.645590:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788571944.645595:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788571944.645608:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3092 00000100:00080000:2.0:1788571944.645620:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1788571944.645628:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788571944.645642:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788571944.645662:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488320000/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788571944.645863:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488320128:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788571944.645899:0:2405:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788571944.645905:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488319872:192.168.202.125@tcp:400:kworker.0 00000100:00000400:2.0:1788571945.670015:0:2406:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788571929/real 1788571929] req@ffff94d408c56a00 x1875450488317824/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1788571945 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788571945.670060:0:2406:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff94d408c56a00 x1875450488317824/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1788571945 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788571945.670069:0:2406:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from FULL to DISCONN 00000100:02020000:2.0:1788571945.670073:0:2406:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.202.125@tcp: Connection to MGS (at 192.168.202.125@tcp) was lost; in progress operations using this service will fail 00000100:00080000:2.0:1788571945.670077:0:2406:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.202.125@tcp for MGC192.168.202.125@tcp not replayable, auto-deactivating 00000100:00080000:2.0:1788571945.670079:0:2406:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:2.0:1788571945.670083:0:2406:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff94d408c56a00 x1875450488317824/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1788571945 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788571945.670092:0:2406:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff94d4125b0700 x1875450488318592/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 0 dl 1788571950 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788571945.670109:0:2406:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff94d408c63100 x1875450488319232/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 0 dl 1788571955 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788571945.670118:0:2406:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff94d408c61c00 x1875450488319872/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 0 dl 1788571960 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:2.0:1788571945.670132:0:2406:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:2.0:1788571945.670136:0:2406:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:2.0:1788571945.670137:0:2406:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:2.0:1788571945.670139:0:2406:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00100000:2.0:1788571945.670155:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488319232:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788571945.670160:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488317824:192.168.202.125@tcp:400:kworker.0 00000100:00080000:2.0:1788571945.701023:0:60899:0:(import.c:1598:ptlrpc_invalidate_import_thread()) thread invalidate import MGC192.168.202.125@tcp to MGS@192.168.202.125@tcp 00000100:00100000:2.0:1788571945.701038:0:60899:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:2.0:1788571945.701042:0:60899:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00010000:00010000:2.0:1788571945.701062:0:60899:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.125@tcp lock: ffff94d4089a2000/0x1e260726fbbbfc27 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110e400000000 nid: local remote: 0xec901abfeb4dc088 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788571945.701074:0:60899:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.125@tcp lock: ffff94d4089a2000/0x1e260726fbbbfc27 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110f400000000 nid: local remote: 0xec901abfeb4dc088 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788571945.701081:0:60899:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:2.0:1788571945.701102:0:60899:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788571945.701113:0:60899:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.125@tcp lock: ffff94d4089a2000/0x1e260726fbbbfc27 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x590f400000000 nid: local remote: 0xec901abfeb4dc088 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571945.701127:0:60899:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.125@tcp lock: ffff94d40481fa00/0x1e260726fbbbfc35 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0xec901abfeb4dc0a4 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788571945.701134:0:60899:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.125@tcp lock: ffff94d40481fa00/0x1e260726fbbbfc35 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0xec901abfeb4dc0a4 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788571945.701139:0:60899:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:2.0:1788571945.701143:0:60899:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788571945.701146:0:60899:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.125@tcp lock: ffff94d40481fa00/0x1e260726fbbbfc35 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0xec901abfeb4dc0a4 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571945.701163:0:60899:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.125@tcp lock: ffff94d4089a2e00/0x1e260726fbbbfc20 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0xec901abfeb4dc081 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788571945.701169:0:60899:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.125@tcp lock: ffff94d4089a2e00/0x1e260726fbbbfc20 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0xec901abfeb4dc081 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788571945.701173:0:60899:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788571945.701176:0:60899:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788571945.701179:0:60899:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.125@tcp lock: ffff94d4089a2e00/0x1e260726fbbbfc20 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0xec901abfeb4dc081 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571945.701189:0:60899:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.125@tcp lock: ffff94d40481e600/0x1e260726fbbbfc2e lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0xec901abfeb4dc09d expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788571945.701195:0:60899:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.125@tcp lock: ffff94d40481e600/0x1e260726fbbbfc2e lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0xec901abfeb4dc09d expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788571945.701200:0:60899:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 00010000:00010000:2.0:1788571945.702748:0:60899:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788571945.702760:0:60899:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.125@tcp lock: ffff94d40481e600/0x1e260726fbbbfc2e lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0xec901abfeb4dc09d expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:2.0:1788571945.702787:0:60899:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from EVICTED to RECOVER 00000100:00080000:2.0:1788571945.702792:0:60899:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from RECOVER to FULL 10000000:01000000:2.0:1788571945.702794:0:60899:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:2.0:1788571945.702797:0:60899:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:02000000:2.0:1788571945.702812:0:60899:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) MGC192.168.202.125@tcp: Connection restored to 192.168.202.125@tcp (at 192.168.202.125@tcp) 10000000:01000000:2.0:1788571947.213035:0:42451:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788571947.213044:0:42451:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-cliir-ffff94d405cc3800 from 89 10000000:01000000:2.0:1788571947.213051:0:42451:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788571947.213102:0:42451:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d41251f000/0x1e260726fbbbfe1f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571947.213114:0:42451:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.125@tcp lock: ffff94d41251f000/0x1e260726fbbbfe1f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571947.213126:0:42451:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.125@tcp lock: ffff94d41251f000/0x1e260726fbbbfe1f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788571947.213154:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488320640:192.168.202.125@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788571947.213284:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408619480 going to sleep for 11 seconds 00000020:01000000:2.0:1788571947.327447:0:42451:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 52-51 (rc=0) 10000000:01000000:2.0:1788571947.327483:0:42451:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.202.125@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:2.0:1788571947.327496:0:42451:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d4089a3600/0x1e260726fbbbfe26 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4dce7a expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571947.327509:0:42451:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.125@tcp lock: ffff94d4089a3600/0x1e260726fbbbfe26 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4dce7a expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788571947.327676:0:42451:0:(mgc_request.c:611:do_requeue()) updating log params 10000000:01000000:2.0:1788571947.327680:0:42451:0:(mgc_request.c:1801:mgc_process_log()) Process log params-ffff94d405cc3800 from 7 10000000:01000000:2.0:1788571947.327684:0:42451:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:2.0:1788571947.327727:0:42451:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d41251e200/0x1e260726fbbbfe2d lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571947.327735:0:42451:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.125@tcp lock: ffff94d41251e200/0x1e260726fbbbfe2d lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788571947.327746:0:42451:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.125@tcp lock: ffff94d41251e200/0x1e260726fbbbfe2d lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788571947.327778:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488321408:192.168.202.125@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788571947.327967:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4086196c0 going to sleep for 11 seconds 00000040:00080000:2.0:1788571947.366431:0:60942:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 64768 count 7 00000100:00100000:2.0:1788571950.801439:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488322048:192.168.202.125@tcp:400:kworker.0 00010000:00010000:2.0:1788571950.811200:0:2405:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 0/0 locks to regular blp list, 1 blwis in pool 00000100:00100000:2.0:1788571950.811245:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488322176:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788571951.815540:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488322432:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788571951.835050:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488322432:192.168.202.125@tcp:400:kworker.0 00000100:00080000:2.0:1788571956.935373:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788571956.935381:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:2.0:1788571956.935511:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488322560:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788571956.935608:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788571956.935615:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488322688:192.168.202.125@tcp:400:kworker.0 00000001:02000400:2.0:1788571960.400132:0:61227:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 00000100:00080000:2.0:1788571967.177567:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788571967.177575:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:2.0:1788571967.177613:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488323200:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788571967.177703:0:2405:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788571967.177710:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488323072:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788571972.293644:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488323328:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788571972.296709:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788571972.296735:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488323456:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788571977.413745:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488323584:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788571977.413858:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788571977.413867:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488323712:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788571982.533645:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488323840:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788571982.533758:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788571982.533766:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488323968:192.168.202.125@tcp:400:kworker.0 00000100:00080000:2.0:1788571987.653560:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788571987.653568:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:2.0:1788571987.653603:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488324224:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788571987.653686:0:2405:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788571987.653693:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488324096:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788571987.656807:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488324224:192.168.202.125@tcp:400:kworker.0 00000100:00080000:2.0:1788571992.773661:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788571992.773675:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00080000:2.0:1788572003.016429:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572003.016436:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:2.0:1788572003.016472:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488324992:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572003.021515:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488324992:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572008.134928:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488325120:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572008.135030:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788572008.135037:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488325248:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572008.141234:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488325248:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572008.141251:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488325120:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572018.384314:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488325760:192.168.202.125@tcp:400:kworker.0 00000100:00080000:2.0:1788572018.425651:0:61629:0:(import.c:1598:ptlrpc_invalidate_import_thread()) thread invalidate import lustre-MDT0000-mdc-ffff94d405cc3800 to lustre-MDT0000_UUID@192.168.202.125@tcp 00000100:00080000:2.0:1788572018.425655:0:61629:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:2.0:1788572018.425659:0:61629:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 81604378628 gen 2 00000100:00100000:2.0:1788572018.425668:0:61629:0:(client.c:3150:ptlrpc_free_committed()) @@@ free request with old gen req@ffff94d408c57480 x1875450488301952/t81604378629(81604378629) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 528/224 e 7 to 0 dl 1788572034 ref 1 fl Complete:RQU/204/0 rc -107/-107 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572018.425686:0:61629:0:(client.c:3150:ptlrpc_free_committed()) @@@ free request with old gen req@ffff94d4125b3100 x1875450488302464/t81604378630(81604378630) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 512/456 e 0 to 0 dl 1788571933 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572018.425699:0:61629:0:(client.c:3150:ptlrpc_free_committed()) @@@ free request with old gen req@ffff94d412927b80 x1875450488304128/t81604378631(81604378631) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 552/448 e 0 to 0 dl 1788571941 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572018.425712:0:61629:0:(client.c:3150:ptlrpc_free_committed()) @@@ free request with old gen req@ffff94d412925f80 x1875450488305024/t81604378633(81604378633) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 600/536 e 0 to 0 dl 1788571941 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572018.425722:0:61629:0:(client.c:3150:ptlrpc_free_committed()) @@@ free request with old gen req@ffff94d408c55c00 x1875450488305920/t81604378635(81604378635) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/664 e 0 to 0 dl 1788571941 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788572018.425730:0:61629:0:(client.c:3150:ptlrpc_free_committed()) @@@ free request with old gen req@ffff94d412927800 x1875450488306176/t81604378636(81604378636) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 600/536 e 0 to 0 dl 1788571941 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572018.425736:0:61629:0:(client.c:3150:ptlrpc_free_committed()) @@@ free request with old gen req@ffff94d408c62300 x1875450488306816/t81604378637(81604378637) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 600/536 e 0 to 0 dl 1788571941 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572018.425743:0:61629:0:(client.c:3150:ptlrpc_free_committed()) @@@ free request with old gen req@ffff94d412925180 x1875450488307456/t81604378638(81604378638) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 600/536 e 0 to 0 dl 1788571941 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572018.425749:0:61629:0:(client.c:3150:ptlrpc_free_committed()) @@@ free request with old gen req@ffff94d4125b0a80 x1875450488308096/t81604378639(81604378639) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 600/536 e 0 to 0 dl 1788571941 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572018.425759:0:61629:0:(client.c:3150:ptlrpc_free_committed()) @@@ free request with old gen req@ffff94d4125b0380 x1875450488308736/t81604378640(81604378640) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 600/536 e 0 to 0 dl 1788571941 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572018.425779:0:61629:0:(client.c:3150:ptlrpc_free_committed()) @@@ free request with old gen req@ffff94d412925500 x1875450488309376/t81604378641(81604378641) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 600/536 e 0 to 0 dl 1788571941 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572018.425789:0:61629:0:(client.c:3150:ptlrpc_free_committed()) @@@ free request with old gen req@ffff94d4125b3480 x1875450488310016/t81604378642(81604378642) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 600/536 e 0 to 0 dl 1788571941 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572018.425799:0:61629:0:(client.c:3150:ptlrpc_free_committed()) @@@ free request with old gen req@ffff94d4125b2680 x1875450488310656/t81604378643(81604378643) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 600/536 e 0 to 0 dl 1788571941 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572018.425811:0:61629:0:(client.c:3150:ptlrpc_free_committed()) @@@ free request with old gen req@ffff94d412925c00 x1875450488311296/t81604378644(81604378644) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 600/536 e 0 to 0 dl 1788571941 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572018.425820:0:61629:0:(client.c:3150:ptlrpc_free_committed()) @@@ free request with old gen req@ffff94d4125b0e00 x1875450488311936/t81604378645(81604378645) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 600/536 e 0 to 0 dl 1788571941 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572018.425827:0:61629:0:(client.c:3150:ptlrpc_free_committed()) @@@ free request with old gen req@ffff94d412924000 x1875450488312576/t81604378646(81604378646) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 600/536 e 0 to 0 dl 1788571941 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572018.425838:0:61629:0:(client.c:3150:ptlrpc_free_committed()) @@@ free request with old gen req@ffff94d412927100 x1875450488313216/t81604378647(81604378647) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 600/536 e 0 to 0 dl 1788571941 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572018.425845:0:61629:0:(client.c:3150:ptlrpc_free_committed()) @@@ free request with old gen req@ffff94d4125b1500 x1875450488313856/t81604378648(81604378648) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 600/536 e 0 to 0 dl 1788571941 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572018.425851:0:61629:0:(client.c:3150:ptlrpc_free_committed()) @@@ free request with old gen req@ffff94d412924380 x1875450488314496/t81604378649(81604378649) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 600/536 e 0 to 0 dl 1788571941 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572018.425857:0:61629:0:(client.c:3150:ptlrpc_free_committed()) @@@ free request with old gen req@ffff94d4125b1180 x1875450488315136/t81604378650(81604378650) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 600/536 e 0 to 0 dl 1788571941 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572018.425863:0:61629:0:(client.c:3150:ptlrpc_free_committed()) @@@ free request with old gen req@ffff94d408c54700 x1875450488315776/t81604378651(81604378651) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 600/536 e 0 to 0 dl 1788571941 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572018.425869:0:61629:0:(client.c:3150:ptlrpc_free_committed()) @@@ free request with old gen req@ffff94d412924700 x1875450488316416/t81604378652(81604378652) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 600/536 e 0 to 0 dl 1788571942 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572018.425875:0:61629:0:(client.c:3150:ptlrpc_free_committed()) @@@ free request with old gen req@ffff94d4125b3800 x1875450488317056/t81604378653(81604378653) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 600/536 e 0 to 0 dl 1788571942 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572018.425881:0:61629:0:(client.c:3150:ptlrpc_free_committed()) @@@ free request with old gen req@ffff94d412924a80 x1875450488317696/t81604378654(81604378654) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 600/536 e 0 to 0 dl 1788571942 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572018.425897:0:61629:0:(client.c:3200:ptlrpc_free_committed()) @@@ free stale open request req@ffff94d412924e00 x1875450488280576/t77309411344(77309411344) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/608 e 0 to 0 dl 1788571963 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788572018.425914:0:61629:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00010000:00010000:2.0:1788572018.426016:0:61629:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ee00/0x1e260726fbbbfc66 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0xec901abfeb4dc486 expref: -99 pid: 60648 timeout: 0 lvb_type: 0 lru_score: 101 lru_type: 1 00010000:00010000:2.0:1788572018.426036:0:61629:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572018.426045:0:61629:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ee00/0x1e260726fbbbfc66 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0xec901abfeb4dc486 expref: -99 pid: 60648 timeout: 0 lvb_type: 0 lru_score: 101 lru_type: 1 00010000:00010000:2.0:1788572018.426487:0:61629:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e800/0x1e260726fbbbfb39 lrc: 3/0,0 mode: PR/PR res: [0x2000061c1:0x17:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0xec901abfeb4dc1b5 expref: -99 pid: 59035 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:2.0:1788572018.426508:0:61629:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481f200/0x1e260726fbbbfb71 lrc: 2/0,0 mode: CR/CR res: [0x2000061c1:0x17:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc1ae expref: -99 pid: 59041 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1788572018.426541:0:61629:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572018.426546:0:61629:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e800/0x1e260726fbbbfb39 lrc: 0/0,0 mode: --/PR res: [0x2000061c1:0x17:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0xec901abfeb4dc1b5 expref: -99 pid: 59035 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:2.0:1788572018.426555:0:61629:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481f200/0x1e260726fbbbfb71 lrc: 3/0,0 mode: CR/CR res: [0x2000061c1:0x17:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0xec901abfeb4dc1ae expref: -99 pid: 59041 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000002:00100000:2.0:1788572018.426609:0:61629:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff94d412924e00 x1875450488280576/t77309411344(77309411344) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/608 e 0 to 0 dl 1788571963 ref 1 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572018.426620:0:61629:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572018.426625:0:61629:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c62300 x1875450488326272/t0(0) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572018.426645:0:61629:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4125d4f00 going to sleep for 0 seconds 00000100:00100000:2.0:1788572018.426652:0:61629:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ll_imp_inval:9bf287c0-f582-4b83-84c1-15e766218620:61629:1875450488326272:192.168.202.125@tcp:35:ll_imp_inval.0 00000002:00100000:2.0:1788572018.426662:0:61629:0:(mdc_request.c:980:mdc_close()) request ffff94d408c62300 failed to send: rc = -108 00000080:00020000:2.0:1788572018.426666:0:61629:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff94d405cc3800: inode [0x2000061c1:0x17:0x0] mdc close failed: rc = -108 00000002:00100000:2.0:1788572018.441328:0:61629:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff94d412924e00 x1875450488280576/t77309411344(77309411344) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/608 e 0 to 0 dl 1788571963 ref 1 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1788572018.441384:0:61629:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572018.441400:0:61629:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481f200/0x1e260726fbbbfb71 lrc: 0/0,0 mode: --/CR res: [0x2000061c1:0x17:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0xec901abfeb4dc1ae expref: -99 pid: 59041 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1788572018.441505:0:61629:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a3400/0x1e260726fbbbfc4a lrc: 3/0,0 mode: PR/PR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0xec901abfeb4dc3de expref: -99 pid: 60561 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:2.0:1788572018.441534:0:61629:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddde00/0x1e260726fbbbfc9e lrc: 2/0,0 mode: CR/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dc551 expref: -99 pid: 60651 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:2.0:1788572018.441556:0:61629:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572018.441561:0:61629:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a3400/0x1e260726fbbbfc4a lrc: 0/0,0 mode: --/PR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0xec901abfeb4dc3de expref: -99 pid: 60561 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:2.0:1788572018.441570:0:61629:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddde00/0x1e260726fbbbfc9e lrc: 3/0,0 mode: CR/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0xec901abfeb4dc551 expref: -99 pid: 60651 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000002:00100000:2.0:1788572018.441625:0:61629:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff94d408c55c00 x1875450488305920/t81604378635(81604378635) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/664 e 0 to 0 dl 1788571941 ref 1 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572018.441636:0:61629:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572018.441642:0:61629:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c62300 x1875450488326528/t0(0) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572018.441667:0:61629:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4125d4f00 going to sleep for 0 seconds 00000100:00100000:2.0:1788572018.441674:0:61629:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ll_imp_inval:9bf287c0-f582-4b83-84c1-15e766218620:61629:1875450488326528:192.168.202.125@tcp:35:ll_imp_inval.0 00000002:00100000:2.0:1788572018.441686:0:61629:0:(mdc_request.c:980:mdc_close()) request ffff94d408c62300 failed to send: rc = -108 00000080:00020000:2.0:1788572018.441690:0:61629:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff94d405cc3800: inode [0x2000061c1:0x1d:0x0] mdc close failed: rc = -108 00000002:00100000:2.0:1788572018.441697:0:61629:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff94d408c55c00 x1875450488305920/t81604378635(81604378635) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/664 e 0 to 0 dl 1788571941 ref 1 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1788572018.441714:0:61629:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572018.441719:0:61629:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddde00/0x1e260726fbbbfc9e lrc: 0/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0xec901abfeb4dc551 expref: -99 pid: 60651 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000100:00080000:2.0:1788572018.445263:0:61629:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from EVICTED to RECOVER 00000100:00080000:2.0:1788572018.445276:0:61629:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from RECOVER to FULL 00000001:02000400:2.0:1788572020.812363:0:61763:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 54 sec 00010000:00010000:2.0:1788572023.429618:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00080000:2.0:1788572023.514615:0:2406:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff94d408c55f80 x1875450488326784/t0(0) o400->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572039 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572023.514640:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488326784:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572023.514659:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488326912:192.168.202.125@tcp:400:kworker.0 00000080:00200000:2.0:1788572024.954020:0:61836:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572024.954051:0:61836:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572024.954071:0:61836:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572024.954088:0:61836:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572024.954096:0:61836:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572024.954104:0:61836:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572024.954108:0:61836:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbbfe3b 00000080:00010000:2.0:1788572024.954117:0:61836:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfe3b from it ffffb68a42377bf8 00010000:00010000:2.0:1788572024.954121:0:61836:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572024.954130:0:61836:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788572024.954147:0:61836:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572024.954159:0:61836:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d119.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), flags=27 00000080:00200000:2.0:1788572024.954163:0:61836:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d119.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), intent=0 00000002:00010000:2.0:1788572024.954176:0:61836:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d119.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1788572024.954247:0:61836:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f600/0x1e260726fbbbfe42 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 61836 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572024.954255:0:61836:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f600/0x1e260726fbbbfe42 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 61836 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572024.954266:0:61836:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f600/0x1e260726fbbbfe42 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 61836 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572024.954294:0:61836:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:61836:1875450488327936:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572024.954384:0:61836:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d412158900 going to sleep for 11 seconds 00000002:00010000:2.0:1788572025.094462:0:61837:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: stripe_dir-1,[0x0:0x0:0x0]) in obj [0x2000061c1:0x1d:0x0], intent: open flags 0504001 00010000:00010000:2.0:1788572025.094549:0:61837:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e400/0x1e260726fbbbfe7a lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 61837 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.094560:0:61837:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e400/0x1e260726fbbbfe7a lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 61837 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572025.094670:0:61837:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572025.094921:0:61837:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c62680 x1875450488330112/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572025.094935:0:61837:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e400/0x1e260726fbbbfe7a lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 61837 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572025.094963:0:61837:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:61837:1875450488330112:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572025.095244:0:61837:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d412158900 going to sleep for 11 seconds 00000080:00200000:2.0:1788572025.201892:0:61838:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572025.201925:0:61838:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572025.201945:0:61838:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1788572025.201961:0:61838:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1788572025.201968:0:61838:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572025.201977:0:61838:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572025.201980:0:61838:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbbfe3b 00000080:00010000:2.0:1788572025.201999:0:61838:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfe3b from it ffffb68a41f6fbf8 00010000:00010000:2.0:1788572025.202004:0:61838:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1788572025.202013:0:61838:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1788572025.202398:0:61838:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572025.202416:0:61838:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d119.replay-single, flags=27 00000080:00200000:2.0:1788572025.202424:0:61838:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572025.202437:0:61838:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=stripe_dir-2, dir=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), file ffff94d4157ef980, open_flags 18800, mode 0 opened 0 00000080:00200000:2.0:1788572025.202444:0:61838:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-2, dir=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), intent=open 00000002:00010000:2.0:1788572025.202460:0:61838:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: stripe_dir-2,[0x0:0x0:0x0]) in obj [0x2000061c1:0x1d:0x0], intent: open flags 0304001 00010000:00010000:2.0:1788572025.202514:0:61838:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e600/0x1e260726fbbbfe81 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 61838 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.202522:0:61838:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e600/0x1e260726fbbbfe81 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 61838 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572025.202534:0:61838:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572025.202546:0:61838:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c63b80 x1875450488331008/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572025.202557:0:61838:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e600/0x1e260726fbbbfe81 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 61838 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572025.202580:0:61838:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:61838:1875450488331008:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572025.202749:0:61838:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d412158900 going to sleep for 11 seconds 00000100:00100000:2.0:1788572025.220528:0:61838:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:61838:1875450488331776:192.168.202.125@tcp:35:lfs.0 00000100:00100000:2.0:1788572025.220602:0:61838:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d412158900 going to sleep for 11 seconds 00000080:00200000:2.0:1788572025.302465:0:61840:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572025.302501:0:61840:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572025.302521:0:61840:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00010000:00010000:2.0:1788572025.302538:0:61840:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00000080:00200000:2.0:1788572025.302546:0:61840:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572025.302554:0:61840:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572025.302558:0:61840:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbbfe3b 00000080:00010000:2.0:1788572025.302566:0:61840:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfe3b from it ffffb68a40d4bbf8 00010000:00010000:2.0:1788572025.302571:0:61840:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00010000:00010000:2.0:1788572025.302581:0:61840:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 0 00000080:00200000:2.0:1788572025.302596:0:61840:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572025.302603:0:61840:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d119.replay-single, flags=27 00000080:00200000:2.0:1788572025.302610:0:61840:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572025.302622:0:61840:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=stripe_dir-4, dir=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), file ffff94d405b41080, open_flags 18800, mode 0 opened 0 00000080:00200000:2.0:1788572025.302628:0:61840:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-4, dir=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), intent=open 00000002:00010000:2.0:1788572025.302641:0:61840:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: stripe_dir-4,[0x0:0x0:0x0]) in obj [0x2000061c1:0x1d:0x0], intent: open flags 0304001 00010000:00010000:2.0:1788572025.302693:0:61840:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbbfeb9 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 61840 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.302701:0:61840:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbbfeb9 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 61840 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572025.302713:0:61840:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572025.302725:0:61840:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c63b80 x1875450488333824/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572025.302734:0:61840:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbbfeb9 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 61840 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572025.302755:0:61840:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:61840:1875450488333824:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572025.303170:0:61840:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d412158900 going to sleep for 11 seconds 00000080:00200000:2.0:1788572025.354758:0:61841:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572025.354920:0:61841:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572025.354939:0:61841:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 0 00010000:00010000:2.0:1788572025.354954:0:61841:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 0 00000080:00200000:2.0:1788572025.354963:0:61841:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572025.354971:0:61841:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572025.354975:0:61841:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbbfe3b 00000080:00010000:2.0:1788572025.355081:0:61841:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfe3b from it ffffb68a41f77bf8 00010000:00010000:2.0:1788572025.355088:0:61841:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 0 00010000:00010000:2.0:1788572025.355098:0:61841:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:2.0:1788572025.355113:0:61841:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572025.355121:0:61841:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d119.replay-single, flags=27 00000080:00200000:2.0:1788572025.355127:0:61841:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572025.355139:0:61841:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=stripe_dir-5, dir=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), file ffff94d416e7f080, open_flags 18800, mode 0 opened 0 00000080:00200000:2.0:1788572025.355144:0:61841:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-5, dir=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), intent=open 00000002:00010000:2.0:1788572025.355158:0:61841:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: stripe_dir-5,[0x0:0x0:0x0]) in obj [0x2000061c1:0x1d:0x0], intent: open flags 0304001 00010000:00010000:2.0:1788572025.355391:0:61841:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbbfed5 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 61841 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.355415:0:61841:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbbfed5 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 61841 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572025.355428:0:61841:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572025.355442:0:61841:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c62680 x1875450488335232/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572025.355452:0:61841:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbbfed5 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 61841 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572025.355476:0:61841:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:61841:1875450488335232:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572025.355804:0:61841:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d412158900 going to sleep for 11 seconds 00000100:00100000:2.0:1788572025.424701:0:61842:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d412926a00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:61842:1875450488337408:192.168.202.125@tcp:35:lfs.0 00000100:00100000:2.0:1788572025.424786:0:61842:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d412158900 going to sleep for 11 seconds 00000100:00100000:2.0:1788572025.427292:0:61842:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d412926a00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:61842:1875450488337408:192.168.202.125@tcp:35:lfs.0 00000002:00100000:2.0:1788572025.427326:0:61842:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff94d412926680 x1875450488337152/t85899345946(85899345946) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 616/696 e 0 to 0 dl 1788572041 ref 2 fl Complete:RQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572025.427412:0:61842:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x2000061c1:0x27:0x0](ffff94d4109e3a88), inode mode 41ed mask 44 00000080:00200008:2.0:1788572025.427448:0:61842:0:(file.c:479:ll_file_release()) START file stripe_dir-6:[0x2000061c1:0x27:0x0](ffff94d4109e3a88), flags 304000 00000080:00200080:2.0:1788572025.427487:0:61842:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x2000061c1:0x27:0x0](ffff94d4109e3a88) cmd=c03066f1 arg=17dc3e0 00000100:00100000:2.0:1788572025.427533:0:61842:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:61842:1875450488337536:192.168.202.125@tcp:33:lfs.0 00000100:00100000:2.0:1788572025.427597:0:61842:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d412158900 going to sleep for 11 seconds 00000100:00100000:2.0:1788572025.432877:0:61842:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:61842:1875450488337536:192.168.202.125@tcp:33:lfs.0 00000080:00200008:2.0:1788572025.433068:0:61842:0:(file.c:479:ll_file_release()) START file stripe_dir-6:[0x2000061c1:0x27:0x0](ffff94d4109e3a88), flags 504000 00000002:00100000:2.0:1788572025.433114:0:61842:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff94d412926300 x1875450488336640/t85899345945(85899345945) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 616/696 e 0 to 0 dl 1788572041 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572025.433126:0:61842:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572025.433131:0:61842:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c62300 x1875450488337792/t0(0) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572025.433153:0:61842:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:61842:1875450488337792:192.168.202.125@tcp:35:lfs.0 00000100:00100000:2.0:1788572025.433252:0:61842:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572025.435630:0:61842:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:61842:1875450488337792:192.168.202.125@tcp:35:lfs.0 00000002:00100000:2.0:1788572025.435677:0:61842:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff94d412926300 x1875450488336640/t85899345945(85899345945) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 616/696 e 0 to 0 dl 1788572041 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572025.494346:0:61844:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572025.494377:0:61844:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572025.494414:0:61844:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:2.0:1788572025.494430:0:61844:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200000:2.0:1788572025.494438:0:61844:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572025.494448:0:61844:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572025.494451:0:61844:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbbfe3b 00000080:00010000:2.0:1788572025.494459:0:61844:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfe3b from it ffffb68a41f7fbf8 00010000:00010000:2.0:1788572025.494463:0:61844:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:2.0:1788572025.494472:0:61844:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00200000:2.0:1788572025.494492:0:61844:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572025.494499:0:61844:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d119.replay-single, flags=27 00000080:00200000:2.0:1788572025.494506:0:61844:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572025.494517:0:61844:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=stripe_dir-8, dir=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), file ffff94d4122bb980, open_flags 18800, mode 0 opened 0 00000080:00200000:2.0:1788572025.494523:0:61844:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-8, dir=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), intent=open 00000002:00010000:2.0:1788572025.494536:0:61844:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: stripe_dir-8,[0x0:0x0:0x0]) in obj [0x2000061c1:0x1d:0x0], intent: open flags 0304001 00010000:00010000:2.0:1788572025.494685:0:61844:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbbff29 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 61844 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.494694:0:61844:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbbff29 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 61844 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572025.494707:0:61844:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572025.494719:0:61844:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c62300 x1875450488339456/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572025.494729:0:61844:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbbff29 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 61844 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572025.494751:0:61844:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:61844:1875450488339456:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572025.495064:0:61844:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572025.501453:0:61844:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:61844:1875450488339456:192.168.202.125@tcp:101:lfs.0 00010000:00010000:2.0:1788572025.501480:0:61844:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbbff29 lrc: 4/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 61844 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.501495:0:61844:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbbff29 lrc: 4/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 61844 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.501507:0:61844:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbbff29 lrc: 4/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 61844 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.501514:0:61844:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbbff29 lrc: 3/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 61844 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.501523:0:61844:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbbff29 lrc: 4/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 61844 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.501531:0:61844:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d41251ec00 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:2.0:1788572025.501536:0:61844:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbbff29 lrc: 4/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 61844 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.501544:0:61844:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbbff29 lrc: 4/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 61844 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.501551:0:61844:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbbff29 lrc: 5/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 61844 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.501558:0:61844:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572025.501565:0:61844:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbbff29 lrc: 3/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 61844 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.501572:0:61844:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbbff29 lrc: 0/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 61844 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572025.501592:0:61844:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff94d408c62300 x1875450488339456/t85899345953(85899345953) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 656/696 e 0 to 0 dl 1788572041 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572025.501605:0:61844:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d408c62300 x1875450488339456/t85899345953(85899345953) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 656/696 e 0 to 0 dl 1788572041 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572025.501622:0:61844:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x2000061c1:0x1d:0x0] suppgids 0 -1: rc 0 00000002:00010000:2.0:1788572025.501681:0:61844:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000090a0:0xb:0x0]) in obj [0x2000090a0:0xb:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572025.501731:0:61844:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e600/0x1e260726fbbbff30 lrc: 3/1,0 mode: --/CR res: [0x2000090a0:0xb:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 61844 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.501739:0:61844:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e600/0x1e260726fbbbff30 lrc: 3/1,0 mode: --/CR res: [0x2000090a0:0xb:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 61844 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.501748:0:61844:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e600/0x1e260726fbbbff30 lrc: 3/1,0 mode: --/CR res: [0x2000090a0:0xb:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 61844 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572025.501769:0:61844:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:61844:1875450488339584:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572025.501861:0:61844:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572025.712152:0:61847:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:61847:1875450488343680:192.168.202.125@tcp:101:lfs.0 00010000:00010000:2.0:1788572025.712180:0:61847:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408dddc00/0x1e260726fbbbff7d lrc: 4/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 61847 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.712195:0:61847:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408dddc00/0x1e260726fbbbff7d lrc: 4/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 61847 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.712216:0:61847:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408dddc00/0x1e260726fbbbff7d lrc: 4/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 61847 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.712225:0:61847:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408dddc00/0x1e260726fbbbff7d lrc: 3/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 61847 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.712235:0:61847:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408dddc00/0x1e260726fbbbff7d lrc: 4/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 61847 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.712245:0:61847:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d408dddc00 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:2.0:1788572025.712250:0:61847:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408dddc00/0x1e260726fbbbff7d lrc: 4/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 61847 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.713065:0:61847:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408dddc00/0x1e260726fbbbff7d lrc: 4/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 61847 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.713078:0:61847:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408dddc00/0x1e260726fbbbff7d lrc: 5/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 61847 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.713086:0:61847:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572025.713094:0:61847:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408dddc00/0x1e260726fbbbff7d lrc: 3/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 61847 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.713101:0:61847:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408dddc00/0x1e260726fbbbff7d lrc: 0/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 61847 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572025.713126:0:61847:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff94d408c54000 x1875450488343680/t85899345965(85899345965) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 656/696 e 0 to 0 dl 1788572041 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572025.713139:0:61847:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d408c54000 x1875450488343680/t85899345965(85899345965) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 656/696 e 0 to 0 dl 1788572041 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572025.713156:0:61847:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x2000061c1:0x1d:0x0] suppgids 0 -1: rc 0 00000002:00010000:2.0:1788572025.713534:0:61847:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000090a0:0xe:0x0]) in obj [0x2000090a0:0xe:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572025.713723:0:61847:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e000/0x1e260726fbbbff84 lrc: 3/1,0 mode: --/CR res: [0x2000090a0:0xe:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 61847 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.713742:0:61847:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e000/0x1e260726fbbbff84 lrc: 3/1,0 mode: --/CR res: [0x2000090a0:0xe:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 61847 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.713837:0:61847:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e000/0x1e260726fbbbff84 lrc: 3/1,0 mode: --/CR res: [0x2000090a0:0xe:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 61847 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572025.713863:0:61847:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:61847:1875450488343808:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572025.715579:0:61847:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572025.724165:0:61847:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:61847:1875450488343808:192.168.202.125@tcp:101:lfs.0 00010000:00010000:2.0:1788572025.724217:0:61847:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e000/0x1e260726fbbbff84 lrc: 4/1,0 mode: --/CR res: [0x2000090a0:0xe:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd477 expref: -99 pid: 61847 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.724231:0:61847:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e000/0x1e260726fbbbff84 lrc: 4/1,0 mode: --/PR res: [0x2000090a0:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd477 expref: -99 pid: 61847 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.724242:0:61847:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e000/0x1e260726fbbbff84 lrc: 4/1,0 mode: PR/PR res: [0x2000090a0:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd477 expref: -99 pid: 61847 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.724253:0:61847:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e000/0x1e260726fbbbff84 lrc: 4/1,0 mode: PR/PR res: [0x2000090a0:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd477 expref: -99 pid: 61847 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.724842:0:61847:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e000/0x1e260726fbbbff84 lrc: 5/2,0 mode: PR/PR res: [0x2000090a0:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd477 expref: -99 pid: 61847 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.724851:0:61847:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e000/0x1e260726fbbbff84 lrc: 5/2,0 mode: PR/PR res: [0x2000090a0:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd477 expref: -99 pid: 61847 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.724858:0:61847:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e000/0x1e260726fbbbff84 lrc: 4/1,0 mode: PR/PR res: [0x2000090a0:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd477 expref: -99 pid: 61847 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572025.724876:0:61847:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408c62a00 x1875450488343808/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/640 e 0 to 0 dl 1788572041 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572025.724889:0:61847:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e000/0x1e260726fbbbff84 lrc: 3/1,0 mode: PR/PR res: [0x2000090a0:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd477 expref: -99 pid: 61847 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.724901:0:61847:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589971616/14 (0 0) 00010000:00010000:2.0:1788572025.724909:0:61847:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e000/0x1e260726fbbbff84 lrc: 3/1,0 mode: PR/PR res: [0x2000090a0:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd477 expref: -99 pid: 61847 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.724917:0:61847:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e000/0x1e260726fbbbff84 lrc: 2/0,0 mode: PR/PR res: [0x2000090a0:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd477 expref: -99 pid: 61847 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:2.0:1788572025.724947:0:61847:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2400090a1:0xe:0x0]) in obj [0x2400090a1:0xe:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572025.725110:0:61847:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d41251f800/0x1e260726fbbbff8b lrc: 3/1,0 mode: --/CR res: [0x2400090a1:0xe:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 61847 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.725118:0:61847:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d41251f800/0x1e260726fbbbff8b lrc: 3/1,0 mode: --/CR res: [0x2400090a1:0xe:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 61847 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.725127:0:61847:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d41251f800/0x1e260726fbbbff8b lrc: 3/1,0 mode: --/CR res: [0x2400090a1:0xe:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 61847 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572025.725148:0:61847:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:61847:1875450488343936:192.168.202.125@tcp:101:lfs.0 00000080:00200000:2.0:1788572025.900561:0:61850:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572025.900593:0:61850:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572025.900612:0:61850:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00010000:00010000:2.0:1788572025.900634:0:61850:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000080:00200000:2.0:1788572025.900642:0:61850:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572025.900649:0:61850:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572025.900653:0:61850:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbbfe3b 00000080:00010000:2.0:1788572025.900660:0:61850:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfe3b from it ffffb68a41f77bf8 00010000:00010000:2.0:1788572025.900664:0:61850:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00010000:00010000:2.0:1788572025.900673:0:61850:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00000080:00200000:2.0:1788572025.900688:0:61850:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572025.900696:0:61850:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d119.replay-single, flags=27 00000080:00200000:2.0:1788572025.900701:0:61850:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572025.900712:0:61850:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=stripe_dir-14, dir=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), file ffff94d408f79c80, open_flags 18800, mode 0 opened 0 00000080:00200000:2.0:1788572025.900718:0:61850:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-14, dir=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), intent=open 00000002:00010000:2.0:1788572025.900732:0:61850:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: stripe_dir-14,[0x0:0x0:0x0]) in obj [0x2000061c1:0x1d:0x0], intent: open flags 0304001 00010000:00010000:2.0:1788572025.900784:0:61850:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbbffd1 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 61850 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.900792:0:61850:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbbffd1 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 61850 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572025.900803:0:61850:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572025.900815:0:61850:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c63800 x1875450488347904/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572025.900826:0:61850:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbbffd1 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 61850 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572025.900849:0:61850:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:61850:1875450488347904:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572025.901747:0:61850:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572025.905766:0:61850:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:61850:1875450488347904:192.168.202.125@tcp:101:lfs.0 00010000:00010000:2.0:1788572025.905792:0:61850:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbbffd1 lrc: 4/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 61850 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.905804:0:61850:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbbffd1 lrc: 4/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 61850 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.905811:0:61850:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbbffd1 lrc: 4/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 61850 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.905819:0:61850:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbbffd1 lrc: 3/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 61850 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.905827:0:61850:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbbffd1 lrc: 4/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 61850 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.905834:0:61850:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d41251ec00 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:2.0:1788572025.905838:0:61850:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbbffd1 lrc: 4/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 61850 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.905846:0:61850:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbbffd1 lrc: 4/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 61850 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.905854:0:61850:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbbffd1 lrc: 5/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 61850 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.905862:0:61850:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572025.905868:0:61850:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbbffd1 lrc: 3/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 61850 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.905875:0:61850:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbbffd1 lrc: 0/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 61850 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572025.905896:0:61850:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff94d408c63800 x1875450488347904/t85899345977(85899345977) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 656/696 e 0 to 0 dl 1788572041 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572025.905910:0:61850:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d408c63800 x1875450488347904/t85899345977(85899345977) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 656/696 e 0 to 0 dl 1788572041 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572025.905928:0:61850:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x2000061c1:0x1d:0x0] suppgids 0 -1: rc 0 00000002:00010000:2.0:1788572025.906055:0:61850:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000090a0:0x11:0x0]) in obj [0x2000090a0:0x11:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572025.906176:0:61850:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ea00/0x1e260726fbbbffd8 lrc: 3/1,0 mode: --/CR res: [0x2000090a0:0x11:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 61850 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.906546:0:61850:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ea00/0x1e260726fbbbffd8 lrc: 3/1,0 mode: --/CR res: [0x2000090a0:0x11:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 61850 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.906556:0:61850:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ea00/0x1e260726fbbbffd8 lrc: 3/1,0 mode: --/CR res: [0x2000090a0:0x11:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 61850 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572025.906577:0:61850:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:61850:1875450488348032:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572025.906682:0:61850:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572025.910775:0:61850:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:61850:1875450488348032:192.168.202.125@tcp:101:lfs.0 00010000:00010000:2.0:1788572025.910802:0:61850:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ea00/0x1e260726fbbbffd8 lrc: 4/1,0 mode: --/CR res: [0x2000090a0:0x11:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd573 expref: -99 pid: 61850 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.910820:0:61850:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ea00/0x1e260726fbbbffd8 lrc: 4/1,0 mode: --/PR res: [0x2000090a0:0x11:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd573 expref: -99 pid: 61850 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.910832:0:61850:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ea00/0x1e260726fbbbffd8 lrc: 4/1,0 mode: PR/PR res: [0x2000090a0:0x11:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd573 expref: -99 pid: 61850 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.910843:0:61850:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ea00/0x1e260726fbbbffd8 lrc: 4/1,0 mode: PR/PR res: [0x2000090a0:0x11:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd573 expref: -99 pid: 61850 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.910856:0:61850:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ea00/0x1e260726fbbbffd8 lrc: 5/2,0 mode: PR/PR res: [0x2000090a0:0x11:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd573 expref: -99 pid: 61850 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.910863:0:61850:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ea00/0x1e260726fbbbffd8 lrc: 5/2,0 mode: PR/PR res: [0x2000090a0:0x11:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd573 expref: -99 pid: 61850 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.910870:0:61850:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ea00/0x1e260726fbbbffd8 lrc: 4/1,0 mode: PR/PR res: [0x2000090a0:0x11:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd573 expref: -99 pid: 61850 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572025.910882:0:61850:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408c62d80 x1875450488348032/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/640 e 0 to 0 dl 1788572041 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572025.910896:0:61850:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ea00/0x1e260726fbbbffd8 lrc: 3/1,0 mode: PR/PR res: [0x2000090a0:0x11:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd573 expref: -99 pid: 61850 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.910910:0:61850:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589971616/17 (0 0) 00010000:00010000:2.0:1788572025.910918:0:61850:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ea00/0x1e260726fbbbffd8 lrc: 3/1,0 mode: PR/PR res: [0x2000090a0:0x11:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd573 expref: -99 pid: 61850 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.910932:0:61850:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ea00/0x1e260726fbbbffd8 lrc: 2/0,0 mode: PR/PR res: [0x2000090a0:0x11:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd573 expref: -99 pid: 61850 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:2.0:1788572025.910962:0:61850:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2400090a1:0x11:0x0]) in obj [0x2400090a1:0x11:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572025.911075:0:61850:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d41251e800/0x1e260726fbbbffdf lrc: 3/1,0 mode: --/CR res: [0x2400090a1:0x11:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 61850 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.911084:0:61850:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d41251e800/0x1e260726fbbbffdf lrc: 3/1,0 mode: --/CR res: [0x2400090a1:0x11:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 61850 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.911159:0:61850:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d41251e800/0x1e260726fbbbffdf lrc: 3/1,0 mode: --/CR res: [0x2400090a1:0x11:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 61850 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572025.911698:0:61850:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:61850:1875450488348160:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572025.911826:0:61850:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572025.914944:0:61850:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:61850:1875450488348160:192.168.202.125@tcp:101:lfs.0 00010000:00010000:2.0:1788572025.914969:0:61850:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d41251e800/0x1e260726fbbbffdf lrc: 4/1,0 mode: --/CR res: [0x2400090a1:0x11:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd581 expref: -99 pid: 61850 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.914995:0:61850:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d41251e800/0x1e260726fbbbffdf lrc: 4/1,0 mode: --/PR res: [0x2400090a1:0x11:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd581 expref: -99 pid: 61850 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.915007:0:61850:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d41251e800/0x1e260726fbbbffdf lrc: 4/1,0 mode: PR/PR res: [0x2400090a1:0x11:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd581 expref: -99 pid: 61850 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.915019:0:61850:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d41251e800/0x1e260726fbbbffdf lrc: 4/1,0 mode: PR/PR res: [0x2400090a1:0x11:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd581 expref: -99 pid: 61850 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.915038:0:61850:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d41251e800/0x1e260726fbbbffdf lrc: 5/2,0 mode: PR/PR res: [0x2400090a1:0x11:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd581 expref: -99 pid: 61850 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.915046:0:61850:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d41251e800/0x1e260726fbbbffdf lrc: 5/2,0 mode: PR/PR res: [0x2400090a1:0x11:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd581 expref: -99 pid: 61850 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.915055:0:61850:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d41251e800/0x1e260726fbbbffdf lrc: 4/1,0 mode: PR/PR res: [0x2400090a1:0x11:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd581 expref: -99 pid: 61850 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572025.915067:0:61850:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408c62d80 x1875450488348160/t0(0) o101->lustre-MDT0001-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/640 e 0 to 0 dl 1788572041 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572025.915092:0:61850:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d41251e800/0x1e260726fbbbffdf lrc: 3/1,0 mode: PR/PR res: [0x2400090a1:0x11:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd581 expref: -99 pid: 61850 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.915112:0:61850:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d404a46c00 type 13 mode 4 res 9663713441/17 (0 0) 00010000:00010000:2.0:1788572025.915121:0:61850:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d41251e800/0x1e260726fbbbffdf lrc: 3/1,0 mode: PR/PR res: [0x2400090a1:0x11:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd581 expref: -99 pid: 61850 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.915128:0:61850:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d41251e800/0x1e260726fbbbffdf lrc: 2/0,0 mode: PR/PR res: [0x2400090a1:0x11:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd581 expref: -99 pid: 61850 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572025.915164:0:61850:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x2000061c1:0x2f:0x0], updating i_size 8192 00000080:00200000:2.0:1788572025.915169:0:61850:0:(namei.c:147:ll_iget()) got inode: [0x2000061c1:0x2f:0x0](ffff94d4109e1988): rc = 0 00000080:00010000:2.0:1788572025.915177:0:61850:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000061c1:0x2f:0x0](ffff94d4109e1988) for lock 0x0 00000080:00200008:2.0:1788572025.915244:0:61850:0:(file.c:1090:ll_file_open()) START file stripe_dir-14:[0x2000061c1:0x2f:0x0](ffff94d4109e1988), flags 304000 00000002:00100000:2.0:1788572025.915479:0:61850:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d408c63800 x1875450488347904/t85899345977(85899345977) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 616/696 e 0 to 0 dl 1788572041 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572025.915536:0:61850:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x2000061c1:0x2f:0x0](ffff94d4109e1988), inode mode 41ed mask 44 00000080:00200000:2.0:1788572025.915578:0:61850:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572025.915593:0:61850:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572025.915603:0:61850:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00010000:00010000:2.0:1788572025.915614:0:61850:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00000080:00200000:2.0:1788572025.915620:0:61850:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572025.915625:0:61850:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572025.915628:0:61850:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbbfe3b 00000080:00010000:2.0:1788572025.915632:0:61850:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfe3b from it ffffb68a41f77bf8 00010000:00010000:2.0:1788572025.915635:0:61850:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00010000:00010000:2.0:1788572025.915642:0:61850:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00000080:00200000:2.0:1788572025.915651:0:61850:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572025.915656:0:61850:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d119.replay-single, flags=24 00000080:00200000:2.0:1788572025.915661:0:61850:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572025.915673:0:61850:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=stripe_dir-14, dir=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), file ffff94d408f78900, open_flags 28800, mode 0 opened 0 00000080:00200000:2.0:1788572025.915677:0:61850:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-14, dir=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), intent=open 00000002:00010000:2.0:1788572025.915690:0:61850:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: stripe_dir-14,[0x0:0x0:0x0]) in obj [0x2000061c1:0x1d:0x0], intent: open flags 0504001 00010000:00010000:2.0:1788572025.915733:0:61850:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e400/0x1e260726fbbbffe6 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 61850 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.915740:0:61850:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e400/0x1e260726fbbbffe6 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 61850 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572025.915752:0:61850:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572025.915758:0:61850:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c62d80 x1875450488348416/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572025.915766:0:61850:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e400/0x1e260726fbbbffe6 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 61850 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572025.915785:0:61850:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:61850:1875450488348416:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572025.915945:0:61850:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572025.920055:0:61850:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:61850:1875450488348416:192.168.202.125@tcp:101:lfs.0 00010000:00010000:2.0:1788572025.920082:0:61850:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e400/0x1e260726fbbbffe6 lrc: 4/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 61850 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.920161:0:61850:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e400/0x1e260726fbbbffe6 lrc: 4/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 61850 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.920169:0:61850:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e400/0x1e260726fbbbffe6 lrc: 4/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 61850 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.920176:0:61850:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e400/0x1e260726fbbbffe6 lrc: 3/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 61850 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.920409:0:61850:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e400/0x1e260726fbbbffe6 lrc: 4/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 61850 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.920416:0:61850:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d41251e400 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:2.0:1788572025.920421:0:61850:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e400/0x1e260726fbbbffe6 lrc: 4/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 61850 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.920427:0:61850:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e400/0x1e260726fbbbffe6 lrc: 4/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 61850 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.920434:0:61850:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e400/0x1e260726fbbbffe6 lrc: 5/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 61850 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.920441:0:61850:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572025.920448:0:61850:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e400/0x1e260726fbbbffe6 lrc: 3/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 61850 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.920454:0:61850:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e400/0x1e260726fbbbffe6 lrc: 0/0,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 61850 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572025.920474:0:61850:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff94d408c62d80 x1875450488348416/t85899345978(85899345978) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 656/696 e 0 to 0 dl 1788572041 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572025.920488:0:61850:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d408c62d80 x1875450488348416/t85899345978(85899345978) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 656/696 e 0 to 0 dl 1788572041 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572025.920504:0:61850:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x2000061c1:0x1d:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572025.920545:0:61850:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x2000061c1:0x2f:0x0], updating i_size 4096 00000080:00200000:2.0:1788572025.920548:0:61850:0:(namei.c:147:ll_iget()) got inode: [0x2000061c1:0x2f:0x0](ffff94d4109e1988): rc = 0 00000080:00010000:2.0:1788572025.920554:0:61850:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000061c1:0x2f:0x0](ffff94d4109e1988) for lock 0x0 00000080:00200008:2.0:1788572025.920582:0:61850:0:(file.c:1090:ll_file_open()) START file stripe_dir-14:[0x2000061c1:0x2f:0x0](ffff94d4109e1988), flags 504000 00000002:00100000:2.0:1788572025.920604:0:61850:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d408c62d80 x1875450488348416/t85899345978(85899345978) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 616/696 e 0 to 0 dl 1788572041 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572025.920638:0:61850:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff94d408c62d80 x1875450488348416/t85899345978(85899345978) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 616/696 e 0 to 0 dl 1788572041 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572025.920651:0:61850:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572025.920656:0:61850:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c61180 x1875450488348672/t0(0) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572025.920677:0:61850:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:61850:1875450488348672:192.168.202.125@tcp:35:lfs.0 00000100:00100000:2.0:1788572025.920814:0:61850:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 11 seconds 00000080:00200000:2.0:1788572025.945108:0:61851:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572025.945139:0:61851:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572025.945157:0:61851:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00010000:00010000:2.0:1788572025.945173:0:61851:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00000080:00200000:2.0:1788572025.945425:0:61851:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572025.945434:0:61851:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572025.945437:0:61851:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbbfe3b 00000080:00010000:2.0:1788572025.945446:0:61851:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfe3b from it ffffb68a41f7fbf8 00010000:00010000:2.0:1788572025.945450:0:61851:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00010000:00010000:2.0:1788572025.945459:0:61851:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00000080:00200000:2.0:1788572025.945473:0:61851:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572025.945482:0:61851:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d119.replay-single, flags=27 00000080:00200000:2.0:1788572025.945488:0:61851:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572025.945500:0:61851:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=stripe_dir-15, dir=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), file ffff94d408648900, open_flags 18800, mode 0 opened 0 00000080:00200000:2.0:1788572025.945506:0:61851:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-15, dir=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), intent=open 00000002:00010000:2.0:1788572025.945520:0:61851:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: stripe_dir-15,[0x0:0x0:0x0]) in obj [0x2000061c1:0x1d:0x0], intent: open flags 0304001 00010000:00010000:2.0:1788572025.945571:0:61851:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e400/0x1e260726fbbbffed lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 61851 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.945580:0:61851:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e400/0x1e260726fbbbffed lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 61851 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572025.945592:0:61851:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572025.945604:0:61851:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c61500 x1875450488349312/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572025.945614:0:61851:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e400/0x1e260726fbbbffed lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 61851 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572025.945636:0:61851:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:61851:1875450488349312:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572025.945858:0:61851:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 11 seconds 00000080:00200000:2.0:1788572025.986141:0:61852:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572025.986169:0:61852:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572025.986195:0:61852:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00010000:00010000:2.0:1788572025.986229:0:61852:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00000080:00200000:2.0:1788572025.986238:0:61852:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572025.986245:0:61852:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572025.986248:0:61852:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbbfe3b 00000080:00010000:2.0:1788572025.986509:0:61852:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfe3b from it ffffb68a41f77bf8 00010000:00010000:2.0:1788572025.986514:0:61852:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00010000:00010000:2.0:1788572025.986524:0:61852:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00000080:00200000:2.0:1788572025.986540:0:61852:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572025.986548:0:61852:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d119.replay-single, flags=27 00000080:00200000:2.0:1788572025.986554:0:61852:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572025.986566:0:61852:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=stripe_dir-16, dir=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), file ffff94d402a66a80, open_flags 18800, mode 0 opened 0 00000080:00200000:2.0:1788572025.986573:0:61852:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-16, dir=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), intent=open 00000002:00010000:2.0:1788572025.986588:0:61852:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: stripe_dir-16,[0x0:0x0:0x0]) in obj [0x2000061c1:0x1d:0x0], intent: open flags 0304001 00010000:00010000:2.0:1788572025.986640:0:61852:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbc0009 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 61852 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572025.986648:0:61852:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbc0009 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 61852 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572025.986660:0:61852:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572025.986673:0:61852:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c60000 x1875450488350720/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572025.986682:0:61852:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbc0009 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 61852 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572025.986704:0:61852:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:61852:1875450488350720:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572025.986929:0:61852:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 11 seconds 00000080:00200000:2.0:1788572026.025658:0:61853:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572026.025688:0:61853:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572026.025707:0:61853:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00010000:00010000:2.0:1788572026.025723:0:61853:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00000080:00200000:2.0:1788572026.025732:0:61853:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572026.025739:0:61853:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572026.025743:0:61853:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbbfe3b 00000080:00010000:2.0:1788572026.025750:0:61853:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfe3b from it ffffb68a41e27bf8 00010000:00010000:2.0:1788572026.025754:0:61853:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00010000:00010000:2.0:1788572026.025763:0:61853:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00000080:00200000:2.0:1788572026.025778:0:61853:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572026.025784:0:61853:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d119.replay-single, flags=27 00000080:00200000:2.0:1788572026.025789:0:61853:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572026.025799:0:61853:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=stripe_dir-17, dir=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), file ffff94d4039b5680, open_flags 18800, mode 0 opened 0 00000080:00200000:2.0:1788572026.025805:0:61853:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-17, dir=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), intent=open 00000002:00010000:2.0:1788572026.025818:0:61853:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: stripe_dir-17,[0x0:0x0:0x0]) in obj [0x2000061c1:0x1d:0x0], intent: open flags 0304001 00010000:00010000:2.0:1788572026.025868:0:61853:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e400/0x1e260726fbbc0025 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 61853 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572026.025875:0:61853:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e400/0x1e260726fbbc0025 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 61853 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572026.025892:0:61853:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572026.025904:0:61853:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c61f80 x1875450488352128/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572026.025915:0:61853:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e400/0x1e260726fbbc0025 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 61853 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572026.025935:0:61853:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:61853:1875450488352128:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572026.026240:0:61853:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 11 seconds 00000080:00200000:2.0:1788572026.096242:0:61854:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572026.096382:0:61854:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572026.096413:0:61854:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00010000:00010000:2.0:1788572026.096431:0:61854:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00000080:00200000:2.0:1788572026.096440:0:61854:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572026.096449:0:61854:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572026.096453:0:61854:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbbfe3b 00000080:00010000:2.0:1788572026.096462:0:61854:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfe3b from it ffffb68a41f7fbf8 00010000:00010000:2.0:1788572026.096466:0:61854:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00010000:00010000:2.0:1788572026.096475:0:61854:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00000080:00200000:2.0:1788572026.096493:0:61854:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572026.096502:0:61854:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d119.replay-single, flags=27 00000080:00200000:2.0:1788572026.096509:0:61854:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572026.096523:0:61854:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=stripe_dir-18, dir=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), file ffff94d4121b9980, open_flags 18800, mode 0 opened 0 00000080:00200000:2.0:1788572026.096531:0:61854:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-18, dir=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88), intent=open 00000002:00010000:2.0:1788572026.096548:0:61854:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: stripe_dir-18,[0x0:0x0:0x0]) in obj [0x2000061c1:0x1d:0x0], intent: open flags 0304001 00010000:00010000:2.0:1788572026.096607:0:61854:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbc0041 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 61854 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572026.096617:0:61854:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbc0041 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 61854 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572026.096631:0:61854:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572026.096644:0:61854:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c63480 x1875450488353536/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572026.096653:0:61854:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbc0041 lrc: 3/1,0 mode: --/CR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 61854 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572026.096894:0:61854:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:61854:1875450488353536:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572026.097262:0:61854:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 11 seconds 00010000:00010000:2.0:1788572027.525511:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572029.573713:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572031.621368:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788572033.304167:0:62419:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 120: DNE fail abort should stop both normal and DNE replay ========================================================== 21:33:51 (1788572031) 00000080:00200000:2.0:1788572033.485976:0:62434:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572033.486009:0:62434:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572033.486028:0:62434:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00010000:00010000:2.0:1788572033.486051:0:62434:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00000080:00200000:2.0:1788572033.486060:0:62434:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572033.486068:0:62434:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572033.486072:0:62434:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbbfe3b 00000080:00010000:2.0:1788572033.486081:0:62434:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbbfe3b from it ffffb68a42007b58 00010000:00010000:2.0:1788572033.486085:0:62434:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00010000:00010000:2.0:1788572033.486094:0:62434:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2a00/0x1e260726fbbbfe3b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd087 expref: -99 pid: 61802 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00000080:00200000:2.0:1788572033.486111:0:62434:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572033.486124:0:62434:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d120.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), flags=13 00000080:00200000:2.0:1788572033.486129:0:62434:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d120.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), intent=getattr 00000002:00010000:2.0:1788572033.486141:0:62434:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d120.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572033.486187:0:62434:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbc0079 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 62434 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572033.486195:0:62434:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbc0079 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 62434 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572033.486396:0:62434:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbc0079 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 62434 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572033.669702:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572035.723928:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572037.778829:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572038.854272:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488358144:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572038.854359:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488358400:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572038.854391:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:2.0:1788572038.854397:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488358016:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572038.854425:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488358272:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572038.854447:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488358528:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572038.866828:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488358016:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572038.866861:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488358272:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572038.866873:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488358528:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572038.866885:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488358144:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572038.866897:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488358400:192.168.202.125@tcp:400:kworker.0 00000001:02000400:2.0:1788572039.776474:0:62496:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 00010000:00010000:2.0:1788572039.813835:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1788572039.879774:0:62498:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572039.879803:0:62498:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572039.879821:0:62498:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572039.879837:0:62498:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572039.879852:0:62498:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572039.879859:0:62498:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572039.879863:0:62498:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:2.0:1788572039.879871:0:62498:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a41fbfb58 00010000:00010000:2.0:1788572039.879875:0:62498:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572039.879885:0:62498:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572039.879899:0:62498:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572039.879907:0:62498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=29 00000080:00200000:2.0:1788572039.879912:0:62498:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572039.879933:0:62498:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=stripe_dir-0, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), flags=13 00000080:00200000:2.0:1788572039.879938:0:62498:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-0, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), intent=getattr 00000002:00010000:2.0:1788572039.879950:0:62498:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: stripe_dir-0,[0x0:0x0:0x0]) in obj [0x200009871:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572039.879992:0:62498:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbc008e lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 62498 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572039.880001:0:62498:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbc008e lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 62498 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572039.880011:0:62498:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbc008e lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 62498 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572039.880036:0:62498:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:62498:1875450488359168:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572039.880397:0:62498:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416eabb40 going to sleep for 11 seconds 00000080:00200000:2.0:1788572040.199843:0:62503:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572040.199871:0:62503:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572040.199889:0:62503:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:2.0:1788572040.199906:0:62503:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00200000:2.0:1788572040.199914:0:62503:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572040.199922:0:62503:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572040.199925:0:62503:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:2.0:1788572040.199933:0:62503:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a41fd7be0 00010000:00010000:2.0:1788572040.199937:0:62503:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:2.0:1788572040.199945:0:62503:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200000:2.0:1788572040.199960:0:62503:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572040.199967:0:62503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=24 00000080:00200000:2.0:1788572040.199973:0:62503:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572040.199983:0:62503:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=dir-3, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), flags=1536 00000080:00200000:2.0:1788572040.199986:0:62503:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 3 00000080:00200000:2.0:1788572040.199991:0:62503:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 3 00000080:00200000:2.0:1788572040.199996:0:62503:0:(namei.c:2239:do_mkdir()) VFS Op:name=dir-3, dir=[0x200009871:0x1:0x0](ffff94d4034900c8) 00000020:00100000:2.0:1788572040.200055:0:62503:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572040.202297:0:62503:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c61880 x1875450488362496/t0(0) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 552/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572040.202334:0:62503:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job mkdir:9bf287c0-f582-4b83-84c1-15e766218620:62503:1875450488362496:192.168.202.125@tcp:36:mkdir.0 00000100:00100000:2.0:1788572040.202483:0:62503:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416eab6c0 going to sleep for 11 seconds 00000080:00200000:2.0:1788572040.221853:0:62504:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572040.221887:0:62504:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572040.221905:0:62504:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:2.0:1788572040.222127:0:62504:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200000:2.0:1788572040.222138:0:62504:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572040.222146:0:62504:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572040.222150:0:62504:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:2.0:1788572040.222159:0:62504:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a420dfb58 00010000:00010000:2.0:1788572040.222163:0:62504:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:2.0:1788572040.222173:0:62504:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00200000:2.0:1788572040.222192:0:62504:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572040.222370:0:62504:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=29 00000080:00200000:2.0:1788572040.222377:0:62504:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572040.222389:0:62504:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=stripe_dir-3, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), flags=13 00000080:00200000:2.0:1788572040.222393:0:62504:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-3, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), intent=getattr 00000002:00010000:2.0:1788572040.222407:0:62504:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: stripe_dir-3,[0x0:0x0:0x0]) in obj [0x200009871:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572040.222452:0:62504:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fa00/0x1e260726fbbc00db lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62504 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572040.222460:0:62504:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fa00/0x1e260726fbbc00db lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62504 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572040.222469:0:62504:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fa00/0x1e260726fbbc00db lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62504 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572040.222497:0:62504:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:62504:1875450488362624:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572040.222618:0:62504:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416eab600 going to sleep for 11 seconds 00000080:00200000:2.0:1788572040.301423:0:62506:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572040.301458:0:62506:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572040.301477:0:62506:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:2.0:1788572040.301492:0:62506:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000080:00200000:2.0:1788572040.301500:0:62506:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572040.301507:0:62506:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572040.301511:0:62506:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:2.0:1788572040.301519:0:62506:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a41fd7b58 00010000:00010000:2.0:1788572040.301523:0:62506:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:2.0:1788572040.301532:0:62506:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000080:00200000:2.0:1788572040.301550:0:62506:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572040.301558:0:62506:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=29 00000080:00200000:2.0:1788572040.301563:0:62506:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572040.301574:0:62506:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=stripe_dir-4, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), flags=13 00000080:00200000:2.0:1788572040.301579:0:62506:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-4, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), intent=getattr 00000002:00010000:2.0:1788572040.301591:0:62506:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: stripe_dir-4,[0x0:0x0:0x0]) in obj [0x200009871:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572040.301639:0:62506:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbc00f0 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62506 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572040.301647:0:62506:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbc00f0 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62506 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572040.301657:0:62506:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbc00f0 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62506 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572040.301682:0:62506:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:62506:1875450488363520:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572040.301769:0:62506:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416eab840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572040.320265:0:62506:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4125d4240 going to sleep for 11 seconds 00000080:00200000:2.0:1788572040.365791:0:62507:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572040.365819:0:62507:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572040.365838:0:62507:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:2.0:1788572040.365854:0:62507:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:2.0:1788572040.365863:0:62507:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572040.365871:0:62507:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572040.365875:0:62507:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:2.0:1788572040.365883:0:62507:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a41e1fbe0 00010000:00010000:2.0:1788572040.365887:0:62507:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:2.0:1788572040.365897:0:62507:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000080:00200000:2.0:1788572040.365913:0:62507:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572040.365920:0:62507:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=24 00000080:00200000:2.0:1788572040.365926:0:62507:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572040.365938:0:62507:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=dir-5, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), flags=1536 00000080:00200000:2.0:1788572040.365941:0:62507:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 3 00000080:00200000:2.0:1788572040.365947:0:62507:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 3 00000080:00200000:2.0:1788572040.365951:0:62507:0:(namei.c:2239:do_mkdir()) VFS Op:name=dir-5, dir=[0x200009871:0x1:0x0](ffff94d4034900c8) 00000020:00100000:2.0:1788572040.366011:0:62507:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572040.366025:0:62507:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c61c00 x1875450488364288/t0(0) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 552/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572040.367024:0:62507:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job mkdir:9bf287c0-f582-4b83-84c1-15e766218620:62507:1875450488364288:192.168.202.125@tcp:36:mkdir.0 00000100:00100000:2.0:1788572040.367136:0:62507:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416eab3c0 going to sleep for 11 seconds 00000080:00200000:2.0:1788572040.412405:0:62508:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572040.412437:0:62508:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572040.412455:0:62508:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00010000:00010000:2.0:1788572040.412472:0:62508:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000080:00200000:2.0:1788572040.412487:0:62508:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572040.412494:0:62508:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572040.412498:0:62508:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:2.0:1788572040.412506:0:62508:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a41fafb58 00010000:00010000:2.0:1788572040.412510:0:62508:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00010000:00010000:2.0:1788572040.412520:0:62508:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00000080:00200000:2.0:1788572040.412535:0:62508:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572040.412542:0:62508:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=29 00000080:00200000:2.0:1788572040.412548:0:62508:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572040.412558:0:62508:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=stripe_dir-5, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), flags=13 00000080:00200000:2.0:1788572040.412563:0:62508:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-5, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), intent=getattr 00000002:00010000:2.0:1788572040.412576:0:62508:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: stripe_dir-5,[0x0:0x0:0x0]) in obj [0x200009871:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572040.412612:0:62508:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fc00/0x1e260726fbbc0105 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62508 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572040.412618:0:62508:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fc00/0x1e260726fbbc0105 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62508 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572040.412624:0:62508:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fc00/0x1e260726fbbc0105 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62508 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572040.412644:0:62508:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:62508:1875450488364416:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572040.412721:0:62508:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416eab300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572040.427957:0:62508:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:62508:1875450488364544:192.168.202.125@tcp:101:lfs.0 00010000:00010000:2.0:1788572040.427992:0:62508:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd600/0x1e260726fbbc010c lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62508 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572040.428006:0:62508:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd600/0x1e260726fbbc010c lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 62508 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572040.428013:0:62508:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd600/0x1e260726fbbc010c lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 62508 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572040.428020:0:62508:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd600/0x1e260726fbbc010c lrc: 3/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 62508 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572040.428029:0:62508:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd600/0x1e260726fbbc010c lrc: 4/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 62508 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572040.428035:0:62508:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d408ddd600 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:2.0:1788572040.428041:0:62508:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd600/0x1e260726fbbc010c lrc: 4/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 62508 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572040.428056:0:62508:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd600/0x1e260726fbbc010c lrc: 4/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 62508 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572040.428064:0:62508:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd600/0x1e260726fbbc010c lrc: 5/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x20e400000000 pid: 62508 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572040.428072:0:62508:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572040.428078:0:62508:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd600/0x1e260726fbbc010c lrc: 3/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 62508 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572040.428085:0:62508:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddd600/0x1e260726fbbc010c lrc: 0/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 62508 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572040.428108:0:62508:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff94d408c56d80 x1875450488364544/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 584/584 e 0 to 0 dl 1788572056 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572040.428122:0:62508:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200009871:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1788572040.428148:0:62508:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 30 res 8589973617/1 (0 0) 00000080:00200000:2.0:1788572040.428282:0:62508:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572040.428302:0:62508:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572040.428311:0:62508:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00000080:00200000:2.0:1788572040.512085:0:62511:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572040.512113:0:62511:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572040.512132:0:62511:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00010000:00010000:2.0:1788572040.512147:0:62511:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00000080:00200000:2.0:1788572040.512156:0:62511:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572040.512163:0:62511:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572040.512167:0:62511:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:2.0:1788572040.512175:0:62511:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a41fefbe0 00010000:00010000:2.0:1788572040.512405:0:62511:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00010000:00010000:2.0:1788572040.512417:0:62511:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00000080:00200000:2.0:1788572040.512438:0:62511:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572040.512446:0:62511:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=24 00000080:00200000:2.0:1788572040.512451:0:62511:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572040.512464:0:62511:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=dir-7, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), flags=1536 00000080:00200000:2.0:1788572040.512468:0:62511:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 3 00000080:00200000:2.0:1788572040.512472:0:62511:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 3 00000080:00200000:2.0:1788572040.512476:0:62511:0:(namei.c:2239:do_mkdir()) VFS Op:name=dir-7, dir=[0x200009871:0x1:0x0](ffff94d4034900c8) 00000020:00100000:2.0:1788572040.512545:0:62511:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572040.512560:0:62511:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c60000 x1875450488366080/t0(0) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 552/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572040.512584:0:62511:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job mkdir:9bf287c0-f582-4b83-84c1-15e766218620:62511:1875450488366080:192.168.202.125@tcp:36:mkdir.0 00000100:00100000:2.0:1788572040.512894:0:62511:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416eab0c0 going to sleep for 11 seconds 00010000:00010000:2.0:1788572040.581075:0:62514:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 30 res 8589973617/1 (0 0) 00000080:00200000:2.0:1788572040.581148:0:62514:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572040.581163:0:62514:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572040.581173:0:62514:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 50 lru_type: 1 00010000:00010000:2.0:1788572040.581447:0:62514:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 50 lru_type: 1 00000080:00200000:2.0:1788572040.581456:0:62514:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572040.581462:0:62514:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572040.581466:0:62514:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:2.0:1788572040.581471:0:62514:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a4201fbe8 00010000:00010000:2.0:1788572040.581475:0:62514:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 50 lru_type: 1 00010000:00010000:2.0:1788572040.581484:0:62514:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 51 lru_type: 1 00000080:00200000:2.0:1788572040.581500:0:62514:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572040.581508:0:62514:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=16408 00000080:00200000:2.0:1788572040.581513:0:62514:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572040.581526:0:62514:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=stripe_dir-8, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), flags=16392 00000080:00200000:2.0:1788572040.581530:0:62514:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-8, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), intent=getattr 00000002:00010000:2.0:1788572040.581542:0:62514:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: stripe_dir-8,[0x0:0x0:0x0]) in obj [0x200009871:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572040.581580:0:62514:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbc014b lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62514 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572040.581587:0:62514:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbc014b lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62514 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572040.581597:0:62514:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbc014b lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62514 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572040.581619:0:62514:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:62514:1875450488367232:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572040.594400:0:62514:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416eab0c0 going to sleep for 11 seconds 00000080:00200000:2.0:1788572040.754303:0:62518:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572040.754335:0:62518:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572040.754354:0:62518:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 61 lru_type: 1 00010000:00010000:2.0:1788572040.754371:0:62518:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 61 lru_type: 1 00000080:00200000:2.0:1788572040.754380:0:62518:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572040.754388:0:62518:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572040.754391:0:62518:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:2.0:1788572040.754400:0:62518:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a4203fb58 00010000:00010000:2.0:1788572040.754404:0:62518:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 61 lru_type: 1 00010000:00010000:2.0:1788572040.754413:0:62518:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 62 lru_type: 1 00000080:00200000:2.0:1788572040.754429:0:62518:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572040.754437:0:62518:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=29 00000080:00200000:2.0:1788572040.754443:0:62518:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572040.754454:0:62518:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=stripe_dir-10, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), flags=13 00000080:00200000:2.0:1788572040.754458:0:62518:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-10, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), intent=getattr 00000002:00010000:2.0:1788572040.754471:0:62518:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: stripe_dir-10,[0x0:0x0:0x0]) in obj [0x200009871:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572040.754516:0:62518:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbc016e lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62518 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572040.754525:0:62518:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbc016e lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62518 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572040.754534:0:62518:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbc016e lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62518 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572040.754559:0:62518:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:62518:1875450488368896:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572040.754852:0:62518:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416eab300 going to sleep for 11 seconds 00000080:00200008:2.0:1788572040.820434:0:62520:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788572040.820441:0:62520:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572040.820459:0:62520:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572040.820470:0:62520:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 70 lru_type: 1 00010000:00010000:2.0:1788572040.820480:0:62520:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 70 lru_type: 1 00000080:00200000:2.0:1788572040.820487:0:62520:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572040.820492:0:62520:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572040.820495:0:62520:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:2.0:1788572040.820500:0:62520:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a41fd7cb8 00010000:00010000:2.0:1788572040.820503:0:62520:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 70 lru_type: 1 00010000:00010000:2.0:1788572040.820510:0:62520:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 71 lru_type: 1 00000080:00200000:2.0:1788572040.820648:0:62520:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572040.820657:0:62520:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572040.820665:0:62520:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 71 lru_type: 1 00010000:00010000:2.0:1788572040.820673:0:62520:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 71 lru_type: 1 00000080:00200000:2.0:1788572040.820680:0:62520:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572040.820684:0:62520:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572040.820687:0:62520:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:2.0:1788572040.820690:0:62520:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a41fd7bf8 00010000:00010000:2.0:1788572040.820694:0:62520:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 71 lru_type: 1 00010000:00010000:2.0:1788572040.820701:0:62520:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 72 lru_type: 1 00000080:00200000:2.0:1788572040.820710:0:62520:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572040.820716:0:62520:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=267 00000080:00200000:2.0:1788572040.820722:0:62520:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 44 00000080:00200008:2.0:1788572040.820727:0:62520:0:(file.c:1090:ll_file_open()) START file d120.replay-single:[0x200009871:0x1:0x0](ffff94d4034900c8), flags 300000 00000080:00200080:2.0:1788572040.820747:0:62520:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8) cmd=c03066f0 arg=7ffd92b49970 00000080:00200000:2.0:1788572040.820759:0:62520:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8) name=stripe_dir-11 stripe_offset=0 stripe_count=2, hash_type=0 00010000:00010000:2.0:1788572040.820786:0:62520:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089cba00/0x1e260726fbbc0191 lrc: 2/0,0 mode: PR/PR res: [0x200009871:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4ddd22 expref: -99 pid: 62520 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572040.820802:0:62520:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582a00/0x1e260726fbbc0087 lrc: 2/0,0 mode: PR/PR res: [0x200009871:0x1:0x0].0x0 bits 0x11/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd795 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 0 00010000:00010000:2.0:1788572040.820843:0:62520:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089cba00/0x1e260726fbbc0191 lrc: 1/0,0 mode: --/PR res: [0x200009871:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4ddd22 expref: -99 pid: 62520 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572040.820850:0:62520:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572040.820853:0:62520:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089cba00/0x1e260726fbbc0191 lrc: 0/0,0 mode: --/PR res: [0x200009871:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4ddd22 expref: -99 pid: 62520 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788572040.820870:0:62520:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572040.820878:0:62520:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c62d80 x1875450488370304/t0(0) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572040.820899:0:62520:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:62520:1875450488370304:192.168.202.125@tcp:36:lfs.0 00000100:00100000:2.0:1788572040.820977:0:62520:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416eab3c0 going to sleep for 11 seconds 00000080:00200000:2.0:1788572040.837172:0:62521:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572040.837324:0:62521:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572040.837346:0:62521:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 72 lru_type: 1 00010000:00010000:2.0:1788572040.837363:0:62521:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 72 lru_type: 1 00000080:00200000:2.0:1788572040.837372:0:62521:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572040.837381:0:62521:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572040.837385:0:62521:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:2.0:1788572040.837393:0:62521:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a42047be0 00010000:00010000:2.0:1788572040.837396:0:62521:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 72 lru_type: 1 00010000:00010000:2.0:1788572040.837406:0:62521:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 73 lru_type: 1 00000080:00200000:2.0:1788572040.837422:0:62521:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572040.837430:0:62521:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=24 00000080:00200000:2.0:1788572040.837436:0:62521:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572040.837449:0:62521:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=dir-12, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), flags=1536 00000080:00200000:2.0:1788572040.837453:0:62521:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 3 00000080:00200000:2.0:1788572040.837459:0:62521:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 3 00000080:00200000:2.0:1788572040.837464:0:62521:0:(namei.c:2239:do_mkdir()) VFS Op:name=dir-12, dir=[0x200009871:0x1:0x0](ffff94d4034900c8) 00000020:00100000:2.0:1788572040.837528:0:62521:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572040.837544:0:62521:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c61180 x1875450488370560/t0(0) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 552/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572040.837572:0:62521:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job mkdir:9bf287c0-f582-4b83-84c1-15e766218620:62521:1875450488370560:192.168.202.125@tcp:36:mkdir.0 00000100:00100000:2.0:1788572040.837783:0:62521:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416eab840 going to sleep for 11 seconds 00000080:00200000:2.0:1788572040.856741:0:62522:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572040.856773:0:62522:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572040.856791:0:62522:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 73 lru_type: 1 00010000:00010000:2.0:1788572040.856807:0:62522:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 73 lru_type: 1 00000080:00200000:2.0:1788572040.856816:0:62522:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572040.856824:0:62522:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572040.856828:0:62522:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:2.0:1788572040.856835:0:62522:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a4204fb58 00010000:00010000:2.0:1788572040.856840:0:62522:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 73 lru_type: 1 00010000:00010000:2.0:1788572040.856848:0:62522:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 74 lru_type: 1 00000080:00200000:2.0:1788572040.856863:0:62522:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572040.856869:0:62522:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=29 00000080:00200000:2.0:1788572040.856875:0:62522:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572040.856885:0:62522:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=stripe_dir-12, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), flags=13 00000080:00200000:2.0:1788572040.856889:0:62522:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-12, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), intent=getattr 00000002:00010000:2.0:1788572040.856969:0:62522:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: stripe_dir-12,[0x0:0x0:0x0]) in obj [0x200009871:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572040.857018:0:62522:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbc0198 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572040.857026:0:62522:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbc0198 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572040.857046:0:62522:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbc0198 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62522 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572040.857074:0:62522:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:62522:1875450488370688:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572040.857234:0:62522:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416eab600 going to sleep for 11 seconds 00000002:00010000:2.0:1788572040.868329:0:62522:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572040.868342:0:62522:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 75 lru_type: 1 00010000:00010000:2.0:1788572040.868354:0:62522:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 75 lru_type: 1 00000080:00200000:2.0:1788572040.868360:0:62522:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572040.868366:0:62522:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572040.868369:0:62522:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:2.0:1788572040.868375:0:62522:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a4204fb58 00010000:00010000:2.0:1788572040.868378:0:62522:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 75 lru_type: 1 00010000:00010000:2.0:1788572040.868390:0:62522:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 76 lru_type: 1 00000080:00200000:2.0:1788572040.868404:0:62522:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572040.868412:0:62522:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=13 00000080:00200008:2.0:1788572040.868421:0:62522:0:(file.c:6285:ll_getattr_dentry()) START file d120.replay-single:[0x200009871:0x1:0x0](ffff94d4034900c8), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788572040.868426:0:62522:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8),name=d120.replay-single 00000002:00010000:2.0:1788572040.868437:0:62522:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009871:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572040.868442:0:62522:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 30 res 8589973617/1 (0 0) 00010000:00010000:2.0:1788572040.868480:0:62522:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ee00/0x1e260726fbbc01a6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 62522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572040.868487:0:62522:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ee00/0x1e260726fbbc01a6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 62522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572040.868497:0:62522:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ee00/0x1e260726fbbc01a6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 62522 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572040.868518:0:62522:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:62522:1875450488370944:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572040.868597:0:62522:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416eab6c0 going to sleep for 11 seconds 00000080:00200000:2.0:1788572041.029198:0:62525:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572041.029245:0:62525:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572041.029263:0:62525:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 84 lru_type: 1 00010000:00010000:2.0:1788572041.029279:0:62525:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 84 lru_type: 1 00000080:00200000:2.0:1788572041.029292:0:62525:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572041.029300:0:62525:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572041.029303:0:62525:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:2.0:1788572041.029310:0:62525:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a42087be0 00010000:00010000:2.0:1788572041.029314:0:62525:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 84 lru_type: 1 00010000:00010000:2.0:1788572041.029322:0:62525:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 85 lru_type: 1 00000080:00200000:2.0:1788572041.029337:0:62525:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572041.029344:0:62525:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=24 00000080:00200000:2.0:1788572041.029350:0:62525:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572041.029361:0:62525:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=dir-14, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), flags=1536 00000080:00200000:2.0:1788572041.029365:0:62525:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 3 00000080:00200000:2.0:1788572041.029369:0:62525:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 3 00000080:00200000:2.0:1788572041.029374:0:62525:0:(namei.c:2239:do_mkdir()) VFS Op:name=dir-14, dir=[0x200009871:0x1:0x0](ffff94d4034900c8) 00000020:00100000:2.0:1788572041.031813:0:62525:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572041.031831:0:62525:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c60700 x1875450488372352/t0(0) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 552/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572041.031856:0:62525:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job mkdir:9bf287c0-f582-4b83-84c1-15e766218620:62525:1875450488372352:192.168.202.125@tcp:36:mkdir.0 00000100:00100000:2.0:1788572041.032570:0:62525:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416eabb40 going to sleep for 11 seconds 00000080:00200000:2.0:1788572041.141082:0:62528:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572041.141117:0:62528:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572041.141141:0:62528:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 91 lru_type: 1 00010000:00010000:2.0:1788572041.141158:0:62528:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 91 lru_type: 1 00000080:00200000:2.0:1788572041.141165:0:62528:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572041.141174:0:62528:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572041.141178:0:62528:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:2.0:1788572041.141186:0:62528:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a4209fb58 00010000:00010000:2.0:1788572041.141190:0:62528:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 91 lru_type: 1 00010000:00010000:2.0:1788572041.141197:0:62528:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 92 lru_type: 1 00000080:00200000:2.0:1788572041.141529:0:62528:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572041.141537:0:62528:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=29 00000080:00200000:2.0:1788572041.141543:0:62528:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572041.141555:0:62528:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=stripe_dir-15, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), flags=13 00000080:00200000:2.0:1788572041.141559:0:62528:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-15, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), intent=getattr 00000002:00010000:2.0:1788572041.141572:0:62528:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: stripe_dir-15,[0x0:0x0:0x0]) in obj [0x200009871:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572041.141692:0:62528:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbc01d7 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62528 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.141700:0:62528:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbc01d7 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62528 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.141710:0:62528:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbc01d7 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62528 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572041.141739:0:62528:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:62528:1875450488373376:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572041.141865:0:62528:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416eab240 going to sleep for 11 seconds 00000100:00100000:2.0:1788572041.145747:0:62528:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:62528:1875450488373376:192.168.202.125@tcp:101:lfs.0 00010000:00010000:2.0:1788572041.145771:0:62528:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbc01d7 lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62528 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.145785:0:62528:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbc01d7 lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 62528 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.145792:0:62528:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbc01d7 lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 62528 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.145800:0:62528:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbc01d7 lrc: 3/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 62528 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.145809:0:62528:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbc01d7 lrc: 4/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 62528 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.145815:0:62528:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d41251ec00 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:2.0:1788572041.145819:0:62528:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbc01d7 lrc: 4/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 62528 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.145826:0:62528:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbc01d7 lrc: 4/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 62528 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.145833:0:62528:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbc01d7 lrc: 5/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x20e400000000 pid: 62528 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.145840:0:62528:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572041.145848:0:62528:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbc01d7 lrc: 3/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 62528 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.145855:0:62528:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbc01d7 lrc: 0/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 62528 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572041.145872:0:62528:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff94d408c62680 x1875450488373376/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 584/584 e 0 to 0 dl 1788572057 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572041.145887:0:62528:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200009871:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1788572041.145914:0:62528:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 30 res 8589973617/1 (0 0) 00000080:00200000:2.0:1788572041.145981:0:62528:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572041.145996:0:62528:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572041.146005:0:62528:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 92 lru_type: 1 00010000:00010000:2.0:1788572041.146017:0:62528:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 92 lru_type: 1 00000080:00200000:2.0:1788572041.146034:0:62528:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572041.146039:0:62528:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572041.146043:0:62528:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:2.0:1788572041.146048:0:62528:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a4209fbe8 00010000:00010000:2.0:1788572041.146052:0:62528:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 92 lru_type: 1 00010000:00010000:2.0:1788572041.146060:0:62528:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 93 lru_type: 1 00000080:00200000:2.0:1788572041.146073:0:62528:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572041.146080:0:62528:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=16408 00000080:00200000:2.0:1788572041.146086:0:62528:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572041.146097:0:62528:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=stripe_dir-15, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), flags=16392 00000080:00200000:2.0:1788572041.146100:0:62528:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-15, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), intent=getattr 00000002:00010000:2.0:1788572041.146110:0:62528:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: stripe_dir-15,[0x0:0x0:0x0]) in obj [0x200009871:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572041.146145:0:62528:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ee00/0x1e260726fbbc01de lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62528 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.146152:0:62528:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ee00/0x1e260726fbbc01de lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62528 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.146160:0:62528:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ee00/0x1e260726fbbc01de lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62528 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572041.146178:0:62528:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:62528:1875450488373504:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572041.146278:0:62528:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416eab240 going to sleep for 11 seconds 00000100:00100000:2.0:1788572041.152434:0:62528:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:62528:1875450488373504:192.168.202.125@tcp:101:lfs.0 00010000:00010000:2.0:1788572041.152458:0:62528:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ee00/0x1e260726fbbc01de lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62528 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.152472:0:62528:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ee00/0x1e260726fbbc01de lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 62528 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.152480:0:62528:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ee00/0x1e260726fbbc01de lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 62528 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.152488:0:62528:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ee00/0x1e260726fbbc01de lrc: 3/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 62528 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.152497:0:62528:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ee00/0x1e260726fbbc01de lrc: 4/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 62528 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.152504:0:62528:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d41251ee00 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:2.0:1788572041.152509:0:62528:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ee00/0x1e260726fbbc01de lrc: 4/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 62528 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.152517:0:62528:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ee00/0x1e260726fbbc01de lrc: 4/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 62528 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.152524:0:62528:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ee00/0x1e260726fbbc01de lrc: 5/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x20e400000000 pid: 62528 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.152531:0:62528:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572041.152537:0:62528:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ee00/0x1e260726fbbc01de lrc: 3/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 62528 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.152543:0:62528:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ee00/0x1e260726fbbc01de lrc: 0/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 62528 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572041.152562:0:62528:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff94d408c62680 x1875450488373504/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 584/584 e 0 to 0 dl 1788572057 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572041.152577:0:62528:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200009871:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1788572041.152615:0:62528:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 30 res 8589973617/1 (0 0) 00000080:00200000:2.0:1788572041.152688:0:62528:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572041.152703:0:62528:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572041.152712:0:62528:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 93 lru_type: 1 00010000:00010000:2.0:1788572041.152724:0:62528:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 93 lru_type: 1 00000080:00200000:2.0:1788572041.152737:0:62528:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572041.152742:0:62528:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572041.152746:0:62528:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:2.0:1788572041.152752:0:62528:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a4209fb58 00010000:00010000:2.0:1788572041.152756:0:62528:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 93 lru_type: 1 00010000:00010000:2.0:1788572041.152765:0:62528:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 94 lru_type: 1 00000080:00200000:2.0:1788572041.152778:0:62528:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572041.152785:0:62528:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=13 00000080:00200008:2.0:1788572041.152795:0:62528:0:(file.c:6285:ll_getattr_dentry()) START file d120.replay-single:[0x200009871:0x1:0x0](ffff94d4034900c8), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788572041.152799:0:62528:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8),name=d120.replay-single 00000002:00010000:2.0:1788572041.152812:0:62528:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009871:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572041.152817:0:62528:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 30 res 8589973617/1 (0 0) 00010000:00010000:2.0:1788572041.152854:0:62528:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbc01e5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 62528 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.152861:0:62528:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbc01e5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 62528 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.152871:0:62528:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbc01e5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 62528 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572041.152901:0:62528:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:62528:1875450488373632:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572041.152967:0:62528:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416eab240 going to sleep for 11 seconds 00000100:00100000:2.0:1788572041.156530:0:62528:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:62528:1875450488373632:192.168.202.125@tcp:101:lfs.0 00010000:00010000:2.0:1788572041.156557:0:62528:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbc01e5 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4ddefe expref: -99 pid: 62528 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.156573:0:62528:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbc01e5 lrc: 4/1,0 mode: --/PR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4ddefe expref: -99 pid: 62528 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.156580:0:62528:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbc01e5 lrc: 4/1,0 mode: --/PR res: [0x200009871:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4ddefe expref: -99 pid: 62528 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.156590:0:62528:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbc01e5 lrc: 4/1,0 mode: PR/PR res: [0x200009871:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4ddefe expref: -99 pid: 62528 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.156600:0:62528:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbc01e5 lrc: 4/1,0 mode: PR/PR res: [0x200009871:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4ddefe expref: -99 pid: 62528 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.156625:0:62528:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbc01e5 lrc: 5/2,0 mode: PR/PR res: [0x200009871:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4ddefe expref: -99 pid: 62528 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.156633:0:62528:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbc01e5 lrc: 5/2,0 mode: PR/PR res: [0x200009871:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4ddefe expref: -99 pid: 62528 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.156640:0:62528:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbc01e5 lrc: 4/1,0 mode: PR/PR res: [0x200009871:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4ddefe expref: -99 pid: 62528 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572041.156652:0:62528:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408c62680 x1875450488373632/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572057 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572041.156666:0:62528:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbc01e5 lrc: 3/1,0 mode: PR/PR res: [0x200009871:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4ddefe expref: -99 pid: 62528 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.156676:0:62528:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589973617/1 (0 0) 00000080:00200000:2.0:1788572041.156682:0:62528:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572041.156710:0:62528:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200009871:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788572041.156716:0:62528:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200009871:0x1:0x0](ffff94d4034900c8) for lock 0x1e260726fbbc01e5 00000080:00010000:2.0:1788572041.156723:0:62528:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200009871:0x1:0x0](ffff94d4034900c8) 00000080:00010000:2.0:1788572041.156726:0:62528:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc01e5 from it ffffb68a4209fcb8 00010000:00010000:2.0:1788572041.156730:0:62528:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbc01e5 lrc: 3/1,0 mode: PR/PR res: [0x200009871:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4ddefe expref: -99 pid: 62528 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.156738:0:62528:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbc01e5 lrc: 2/0,0 mode: PR/PR res: [0x200009871:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4ddefe expref: -99 pid: 62528 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788572041.157359:0:62528:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788572041.157367:0:62528:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572041.157662:0:62528:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572041.157674:0:62528:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 94 lru_type: 1 00010000:00010000:2.0:1788572041.157687:0:62528:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 94 lru_type: 1 00000080:00200000:2.0:1788572041.157695:0:62528:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572041.157702:0:62528:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572041.157705:0:62528:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:2.0:1788572041.157710:0:62528:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a4209fcb8 00010000:00010000:2.0:1788572041.157714:0:62528:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 94 lru_type: 1 00010000:00010000:2.0:1788572041.157724:0:62528:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 95 lru_type: 1 00000080:00200000:2.0:1788572041.157872:0:62528:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572041.157882:0:62528:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572041.157889:0:62528:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 95 lru_type: 1 00010000:00010000:2.0:1788572041.157898:0:62528:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 95 lru_type: 1 00000080:00200000:2.0:1788572041.157904:0:62528:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572041.157908:0:62528:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572041.157910:0:62528:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:2.0:1788572041.157914:0:62528:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a4209fbf8 00010000:00010000:2.0:1788572041.157917:0:62528:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 95 lru_type: 1 00010000:00010000:2.0:1788572041.157924:0:62528:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 96 lru_type: 1 00000080:00200000:2.0:1788572041.157933:0:62528:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572041.157939:0:62528:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=267 00000080:00200000:2.0:1788572041.157944:0:62528:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 44 00000080:00200008:2.0:1788572041.157950:0:62528:0:(file.c:1090:ll_file_open()) START file d120.replay-single:[0x200009871:0x1:0x0](ffff94d4034900c8), flags 300000 00000080:00200080:2.0:1788572041.157970:0:62528:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8) cmd=c03066f0 arg=7ffcf0f7de70 00000080:00200000:2.0:1788572041.157981:0:62528:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8) name=stripe_dir-15 stripe_offset=0 stripe_count=2, hash_type=0 00010000:00010000:2.0:1788572041.158005:0:62528:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbc01e5 lrc: 2/0,0 mode: PR/PR res: [0x200009871:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4ddefe expref: -99 pid: 62528 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572041.158031:0:62528:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582a00/0x1e260726fbbc0087 lrc: 2/0,0 mode: PR/PR res: [0x200009871:0x1:0x0].0x0 bits 0x11/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd795 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:2.0:1788572041.158071:0:62528:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbc01e5 lrc: 1/0,0 mode: --/PR res: [0x200009871:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4ddefe expref: -99 pid: 62528 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572041.158079:0:62528:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572041.158081:0:62528:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbc01e5 lrc: 0/0,0 mode: --/PR res: [0x200009871:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4ddefe expref: -99 pid: 62528 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788572041.158100:0:62528:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572041.158111:0:62528:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c62680 x1875450488373888/t0(0) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572041.158135:0:62528:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:62528:1875450488373888:192.168.202.125@tcp:36:lfs.0 00000100:00100000:2.0:1788572041.158241:0:62528:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416eab240 going to sleep for 11 seconds 00000100:00100000:2.0:1788572041.171061:0:62528:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:62528:1875450488373888:192.168.202.125@tcp:36:lfs.0 00000080:00200000:2.0:1788572041.171124:0:62528:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200009871:0x21:0x0](ffff94d403433a88) 00000080:00200000:2.0:1788572041.171133:0:62528:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200009871:0x21:0x0], updating i_size 4096 00000080:00200008:2.0:1788572041.171167:0:62528:0:(file.c:479:ll_file_release()) START file d120.replay-single:[0x200009871:0x1:0x0](ffff94d4034900c8), flags 300000 00010000:00010000:2.0:1788572041.171244:0:62528:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583000/0x1e260726fbbc00bf lrc: 2/0,0 mode: CR/CR res: [0x200009871:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd875 expref: -99 pid: 62500 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 0 00000080:00200000:2.0:1788572041.180856:0:62529:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572041.180958:0:62529:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572041.180980:0:62529:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 96 lru_type: 1 00010000:00010000:2.0:1788572041.180996:0:62529:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 96 lru_type: 1 00000080:00200000:2.0:1788572041.181005:0:62529:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572041.181013:0:62529:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572041.181017:0:62529:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:2.0:1788572041.181034:0:62529:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a420a7be0 00010000:00010000:2.0:1788572041.181039:0:62529:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 96 lru_type: 1 00010000:00010000:2.0:1788572041.181048:0:62529:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 97 lru_type: 1 00000080:00200000:2.0:1788572041.181064:0:62529:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572041.181072:0:62529:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=24 00000080:00200000:2.0:1788572041.181078:0:62529:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572041.181090:0:62529:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=dir-16, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), flags=1536 00000080:00200000:2.0:1788572041.181093:0:62529:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 3 00000080:00200000:2.0:1788572041.181098:0:62529:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 3 00000080:00200000:2.0:1788572041.181103:0:62529:0:(namei.c:2239:do_mkdir()) VFS Op:name=dir-16, dir=[0x200009871:0x1:0x0](ffff94d4034900c8) 00000020:00100000:2.0:1788572041.181172:0:62529:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572041.181186:0:62529:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c63b80 x1875450488374144/t0(0) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 552/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572041.181306:0:62529:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job mkdir:9bf287c0-f582-4b83-84c1-15e766218620:62529:1875450488374144:192.168.202.125@tcp:36:mkdir.0 00000100:00100000:2.0:1788572041.181499:0:62529:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416eab240 going to sleep for 11 seconds 00000080:00200000:2.0:1788572041.339916:0:62534:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572041.339945:0:62534:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572041.339964:0:62534:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 109 lru_type: 1 00010000:00010000:2.0:1788572041.339979:0:62534:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 109 lru_type: 1 00000080:00200000:2.0:1788572041.339987:0:62534:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572041.339995:0:62534:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572041.339998:0:62534:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:2.0:1788572041.340006:0:62534:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a420f7b58 00010000:00010000:2.0:1788572041.340010:0:62534:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 109 lru_type: 1 00010000:00010000:2.0:1788572041.340031:0:62534:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 110 lru_type: 1 00000080:00200000:2.0:1788572041.340047:0:62534:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572041.340055:0:62534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=29 00000080:00200000:2.0:1788572041.340060:0:62534:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572041.340072:0:62534:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=stripe_dir-18, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), flags=13 00000080:00200000:2.0:1788572041.340076:0:62534:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-18, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), intent=getattr 00000002:00010000:2.0:1788572041.340089:0:62534:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: stripe_dir-18,[0x0:0x0:0x0]) in obj [0x200009871:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572041.340134:0:62534:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbc0216 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62534 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.340147:0:62534:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbc0216 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62534 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.340157:0:62534:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbc0216 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62534 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572041.340187:0:62534:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:62534:1875450488376064:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572041.340311:0:62534:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416eab780 going to sleep for 11 seconds 00010000:00010000:2.0:1788572041.405761:0:62536:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 30 res 8589973617/1 (0 0) 00000080:00200000:2.0:1788572041.405838:0:62536:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572041.405853:0:62536:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572041.405863:0:62536:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 117 lru_type: 1 00010000:00010000:2.0:1788572041.405950:0:62536:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 117 lru_type: 1 00000080:00200000:2.0:1788572041.405958:0:62536:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572041.405964:0:62536:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572041.405968:0:62536:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:2.0:1788572041.405974:0:62536:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a41fe7b58 00010000:00010000:2.0:1788572041.405977:0:62536:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 117 lru_type: 1 00010000:00010000:2.0:1788572041.405987:0:62536:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 118 lru_type: 1 00000080:00200000:2.0:1788572041.406007:0:62536:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572041.406024:0:62536:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=13 00000080:00200008:2.0:1788572041.406034:0:62536:0:(file.c:6285:ll_getattr_dentry()) START file d120.replay-single:[0x200009871:0x1:0x0](ffff94d4034900c8), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788572041.406039:0:62536:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8),name=d120.replay-single 00000002:00010000:2.0:1788572041.406051:0:62536:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009871:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572041.406056:0:62536:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 30 res 8589973617/1 (0 0) 00010000:00010000:2.0:1788572041.406094:0:62536:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fe00/0x1e260726fbbc0239 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 62536 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.406103:0:62536:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fe00/0x1e260726fbbc0239 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 62536 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.406113:0:62536:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fe00/0x1e260726fbbc0239 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 62536 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572041.406137:0:62536:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:62536:1875450488377216:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572041.406514:0:62536:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416eab900 going to sleep for 11 seconds 00000080:00200000:2.0:1788572041.413040:0:62536:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572041.413055:0:62536:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572041.413064:0:62536:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 119 lru_type: 1 00010000:00010000:2.0:1788572041.413074:0:62536:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 119 lru_type: 1 00000080:00200000:2.0:1788572041.413081:0:62536:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572041.413086:0:62536:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572041.413089:0:62536:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:2.0:1788572041.413093:0:62536:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a41fe7bf8 00010000:00010000:2.0:1788572041.413097:0:62536:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 119 lru_type: 1 00010000:00010000:2.0:1788572041.413105:0:62536:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 120 lru_type: 1 00000080:00200000:2.0:1788572041.413114:0:62536:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572041.413120:0:62536:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=267 00000080:00200000:2.0:1788572041.413126:0:62536:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 44 00000080:00200008:2.0:1788572041.413132:0:62536:0:(file.c:1090:ll_file_open()) START file d120.replay-single:[0x200009871:0x1:0x0](ffff94d4034900c8), flags 300000 00000080:00200080:2.0:1788572041.413154:0:62536:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8) cmd=c03066f0 arg=7ffdf5b5b4e0 00000080:00200000:2.0:1788572041.413167:0:62536:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8) name=stripe_dir-19 stripe_offset=0 stripe_count=2, hash_type=0 00010000:00010000:2.0:1788572041.413526:0:62536:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fe00/0x1e260726fbbc0239 lrc: 2/0,0 mode: PR/PR res: [0x200009871:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4de0da expref: -99 pid: 62536 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572041.413544:0:62536:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582a00/0x1e260726fbbc0087 lrc: 2/0,0 mode: PR/PR res: [0x200009871:0x1:0x0].0x0 bits 0x11/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd795 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:2.0:1788572041.413588:0:62536:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fe00/0x1e260726fbbc0239 lrc: 1/0,0 mode: --/PR res: [0x200009871:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4de0da expref: -99 pid: 62536 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572041.413658:0:62536:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572041.413662:0:62536:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fe00/0x1e260726fbbc0239 lrc: 0/0,0 mode: --/PR res: [0x200009871:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4de0da expref: -99 pid: 62536 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788572041.413682:0:62536:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572041.413694:0:62536:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c63800 x1875450488377472/t0(0) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572041.413718:0:62536:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:62536:1875450488377472:192.168.202.125@tcp:36:lfs.0 00000100:00100000:2.0:1788572041.413837:0:62536:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416eab9c0 going to sleep for 11 seconds 00000080:00200000:2.0:1788572041.441127:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572041.441160:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572041.441180:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 120 lru_type: 1 00010000:00010000:2.0:1788572041.441197:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 120 lru_type: 1 00000080:00200000:2.0:1788572041.441228:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572041.441236:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572041.441240:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:2.0:1788572041.441247:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a40da3bf8 00010000:00010000:2.0:1788572041.441251:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 120 lru_type: 1 00010000:00010000:2.0:1788572041.441261:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 121 lru_type: 1 00000080:00200000:2.0:1788572041.441276:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572041.441294:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=267 00000080:00200000:2.0:1788572041.441302:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 44 00000080:00200008:2.0:1788572041.441309:0:62287:0:(file.c:1090:ll_file_open()) START file d120.replay-single:[0x200009871:0x1:0x0](ffff94d4034900c8), flags 304000 00000080:00200008:2.0:1788572041.441329:0:62287:0:(file.c:6285:ll_getattr_dentry()) START file d120.replay-single:[0x200009871:0x1:0x0](ffff94d4034900c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572041.441335:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8),name=d120.replay-single 00000002:00010000:2.0:1788572041.441346:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009871:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572041.441353:0:62287:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 30 res 8589973617/1 (0 0) 00010000:00010000:2.0:1788572041.443294:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fe00/0x1e260726fbbc0240 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 62287 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.443311:0:62287:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fe00/0x1e260726fbbc0240 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 62287 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.443323:0:62287:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fe00/0x1e260726fbbc0240 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 62287 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572041.443351:0:62287:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job bash:9bf287c0-f582-4b83-84c1-15e766218620:62287:1875450488377600:192.168.202.125@tcp:101:bash.0 00000100:00100000:2.0:1788572041.443746:0:62287:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416eaba80 going to sleep for 11 seconds 00000080:00200000:2.0:1788572041.474818:0:62539:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572041.474848:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572041.474867:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 121 lru_type: 1 00010000:00010000:2.0:1788572041.474883:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 121 lru_type: 1 00000080:00200000:2.0:1788572041.474891:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572041.474899:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572041.474902:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:2.0:1788572041.474909:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a41f8fb58 00010000:00010000:2.0:1788572041.474913:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 121 lru_type: 1 00010000:00010000:2.0:1788572041.474928:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 122 lru_type: 1 00000080:00200000:2.0:1788572041.474943:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572041.474951:0:62539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=24 00000080:00200000:2.0:1788572041.474957:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572041.474967:0:62539:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=dir-0, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), flags=8 00000080:00200000:2.0:1788572041.474971:0:62539:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=dir-0, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), intent=getattr 00000002:00010000:2.0:1788572041.474983:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: dir-0,[0x0:0x0:0x0]) in obj [0x200009871:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572041.475157:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ee00/0x1e260726fbbc0247 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.475166:0:62539:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ee00/0x1e260726fbbc0247 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.475176:0:62539:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ee00/0x1e260726fbbc0247 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572041.475340:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488377856:192.168.202.125@tcp:101:rm.0 00010000:00010000:2.0:1788572041.789488:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbc03ba lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789497:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ec00/0x1e260726fbbc03ba lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789522:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e400/0x1e260726fbbc03c1 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 6 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789529:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e400/0x1e260726fbbc03c1 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789547:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fe00/0x1e260726fbbc03c8 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 7 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789554:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fe00/0x1e260726fbbc03c8 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789570:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ee00/0x1e260726fbbc03cf lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 8 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789577:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ee00/0x1e260726fbbc03cf lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789595:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbc03d6 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 9 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789600:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f400/0x1e260726fbbc03d6 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789614:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fc00/0x1e260726fbbc03dd lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 10 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789620:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fc00/0x1e260726fbbc03dd lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789632:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbc03e4 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 11 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789638:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f200/0x1e260726fbbc03e4 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 11 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789652:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fa00/0x1e260726fbbc03eb lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 12 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789658:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251fa00/0x1e260726fbbc03eb lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789673:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405ffee00/0x1e260726fbbc03f2 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 13 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789679:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405ffee00/0x1e260726fbbc03f2 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 13 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789696:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405ffe400/0x1e260726fbbc03f9 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 14 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789702:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405ffe400/0x1e260726fbbc03f9 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 14 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789721:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405fff200/0x1e260726fbbc0400 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 15 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789727:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405fff200/0x1e260726fbbc0400 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 15 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789810:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405ffec00/0x1e260726fbbc0407 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 16 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789815:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405ffec00/0x1e260726fbbc0407 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 16 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789827:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405fff400/0x1e260726fbbc040e lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 17 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789830:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405fff400/0x1e260726fbbc040e lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 17 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789839:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405fffa00/0x1e260726fbbc0415 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 18 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789843:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405fffa00/0x1e260726fbbc0415 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 18 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789851:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405ffea00/0x1e260726fbbc041c lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 19 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789855:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405ffea00/0x1e260726fbbc041c lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 19 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789866:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405fff600/0x1e260726fbbc0423 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 20 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789872:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405fff600/0x1e260726fbbc0423 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 20 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789888:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405ffe000/0x1e260726fbbc042a lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 21 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789895:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405ffe000/0x1e260726fbbc042a lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 21 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789927:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405ffe600/0x1e260726fbbc0431 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 22 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789932:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405ffe600/0x1e260726fbbc0431 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 22 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789941:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405fffc00/0x1e260726fbbc0438 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 23 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789945:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405fffc00/0x1e260726fbbc0438 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 23 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789954:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405fffe00/0x1e260726fbbc043f lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 24 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789957:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405fffe00/0x1e260726fbbc043f lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 24 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789967:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405fff800/0x1e260726fbbc0446 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 25 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789971:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405fff800/0x1e260726fbbc0446 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 25 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789979:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405ffe200/0x1e260726fbbc044d lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 26 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789983:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405ffe200/0x1e260726fbbc044d lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 26 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.789992:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405ffe800/0x1e260726fbbc0454 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 27 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.790007:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405ffe800/0x1e260726fbbc0454 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 27 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.790029:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405fff000/0x1e260726fbbc045b lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 28 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.790044:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405fff000/0x1e260726fbbc045b lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 28 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.790066:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4013a0c00/0x1e260726fbbc0462 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 29 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.790072:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4013a0c00/0x1e260726fbbc0462 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 29 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.790088:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4013a1400/0x1e260726fbbc0469 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 30 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.790098:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4013a1400/0x1e260726fbbc0469 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 30 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.790116:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4013a1a00/0x1e260726fbbc0470 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 31 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.790130:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4013a1a00/0x1e260726fbbc0470 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 31 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.790145:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4013a0a00/0x1e260726fbbc0477 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 32 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.790150:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4013a0a00/0x1e260726fbbc0477 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 32 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.790158:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4013a1c00/0x1e260726fbbc047e lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 33 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.790161:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4013a1c00/0x1e260726fbbc047e lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 33 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.790169:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4013a1600/0x1e260726fbbc0485 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 34 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.790173:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4013a1600/0x1e260726fbbc0485 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 34 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.790190:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4013a0600/0x1e260726fbbc048c lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 35 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.790196:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4013a0600/0x1e260726fbbc048c lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 35 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.790244:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40148e800/0x1e260726fbbc0493 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 36 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.790251:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40148e800/0x1e260726fbbc0493 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 36 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.790268:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40148fe00/0x1e260726fbbc049a lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 37 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.790274:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40148fe00/0x1e260726fbbc049a lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 37 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823453:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40148fc00/0x1e260726fbbc04a8 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823460:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40148fc00/0x1e260726fbbc04a8 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823473:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40148ee00/0x1e260726fbbc04af lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 6 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823477:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40148ee00/0x1e260726fbbc04af lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823486:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40148f600/0x1e260726fbbc04b6 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 7 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823490:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40148f600/0x1e260726fbbc04b6 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823498:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40148e600/0x1e260726fbbc04bd lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 8 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823502:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40148e600/0x1e260726fbbc04bd lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823510:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40148fe00/0x1e260726fbbc04c4 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 9 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823514:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40148fe00/0x1e260726fbbc04c4 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823522:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40148e800/0x1e260726fbbc04cb lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 10 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823525:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40148e800/0x1e260726fbbc04cb lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823533:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40148f000/0x1e260726fbbc04d2 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 11 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823536:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40148f000/0x1e260726fbbc04d2 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 11 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823544:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40148e000/0x1e260726fbbc04d9 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 12 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823548:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40148e000/0x1e260726fbbc04d9 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823555:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40148e200/0x1e260726fbbc04e0 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 13 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823558:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40148e200/0x1e260726fbbc04e0 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 13 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823566:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40148f800/0x1e260726fbbc04e7 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 14 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823574:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40148f800/0x1e260726fbbc04e7 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 14 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823586:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405109a00/0x1e260726fbbc04ee lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 15 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823589:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405109a00/0x1e260726fbbc04ee lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 15 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823597:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405108200/0x1e260726fbbc04f5 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 16 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823601:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405108200/0x1e260726fbbc04f5 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 16 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823608:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405109400/0x1e260726fbbc04fc lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 17 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823612:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405109400/0x1e260726fbbc04fc lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 17 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823621:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405109000/0x1e260726fbbc0503 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 18 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823624:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405109000/0x1e260726fbbc0503 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 18 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823632:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405108e00/0x1e260726fbbc050a lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 19 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823635:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405108e00/0x1e260726fbbc050a lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 19 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823643:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405109c00/0x1e260726fbbc0511 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 20 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823647:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405109c00/0x1e260726fbbc0511 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 20 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823657:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405108600/0x1e260726fbbc0518 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 21 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823661:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405108600/0x1e260726fbbc0518 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 21 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823669:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405109800/0x1e260726fbbc051f lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 22 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823673:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405109800/0x1e260726fbbc051f lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 22 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823681:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405108000/0x1e260726fbbc0526 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 23 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823685:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405108000/0x1e260726fbbc0526 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 23 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823694:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405108400/0x1e260726fbbc052d lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 24 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823697:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405108400/0x1e260726fbbc052d lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 24 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823706:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405109e00/0x1e260726fbbc0534 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 25 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823709:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405109e00/0x1e260726fbbc0534 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 25 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823717:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405109200/0x1e260726fbbc053b lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 26 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823721:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405109200/0x1e260726fbbc053b lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 26 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823729:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405109600/0x1e260726fbbc0542 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 27 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823740:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405109600/0x1e260726fbbc0542 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 27 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823758:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405108800/0x1e260726fbbc0549 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 28 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823765:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405108800/0x1e260726fbbc0549 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 28 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823782:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405108a00/0x1e260726fbbc0550 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 29 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823789:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405108a00/0x1e260726fbbc0550 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 29 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823805:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405108c00/0x1e260726fbbc0557 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 30 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823809:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d405108c00/0x1e260726fbbc0557 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 30 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823830:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4128eea00/0x1e260726fbbc055e lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 31 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823834:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4128eea00/0x1e260726fbbc055e lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 31 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823846:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4128efa00/0x1e260726fbbc0565 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 32 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823850:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4128efa00/0x1e260726fbbc0565 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 32 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823864:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4128ef400/0x1e260726fbbc056c lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 33 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823868:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4128ef400/0x1e260726fbbc056c lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 33 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823876:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4128eec00/0x1e260726fbbc0573 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 34 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823880:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4128eec00/0x1e260726fbbc0573 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 34 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823891:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4128ef200/0x1e260726fbbc057a lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 35 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823895:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4128ef200/0x1e260726fbbc057a lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 35 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823904:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4128ee400/0x1e260726fbbc0581 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 36 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823908:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4128ee400/0x1e260726fbbc0581 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 36 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823916:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4128eee00/0x1e260726fbbc0588 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 37 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823919:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4128eee00/0x1e260726fbbc0588 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 37 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823928:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4128ef000/0x1e260726fbbc058f lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 38 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823931:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4128ef000/0x1e260726fbbc058f lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 38 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823939:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4128ee800/0x1e260726fbbc0596 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 39 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823943:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4128ee800/0x1e260726fbbc0596 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 39 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823952:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4128ee200/0x1e260726fbbc059d lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 40 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823955:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4128ee200/0x1e260726fbbc059d lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 40 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823963:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4128ef800/0x1e260726fbbc05a4 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 41 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823967:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4128ef800/0x1e260726fbbc05a4 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 41 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823975:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4128efe00/0x1e260726fbbc05ab lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 42 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823979:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4128efe00/0x1e260726fbbc05ab lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 42 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823987:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4128efc00/0x1e260726fbbc05b2 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 43 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.823990:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4128efc00/0x1e260726fbbc05b2 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 43 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824013:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4128ee600/0x1e260726fbbc05b9 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 44 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824016:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4128ee600/0x1e260726fbbc05b9 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 44 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824027:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4128ee000/0x1e260726fbbc05c0 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 45 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824031:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4128ee000/0x1e260726fbbc05c0 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 45 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824040:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4128ef600/0x1e260726fbbc05c7 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 46 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824043:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4128ef600/0x1e260726fbbc05c7 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 46 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824059:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408cf0000/0x1e260726fbbc05ce lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 47 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824063:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408cf0000/0x1e260726fbbc05ce lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 47 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824075:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408cf1600/0x1e260726fbbc05d5 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 48 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824078:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408cf1600/0x1e260726fbbc05d5 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 48 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824086:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408cf0a00/0x1e260726fbbc05dc lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 49 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824090:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408cf0a00/0x1e260726fbbc05dc lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 49 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824099:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408cf1a00/0x1e260726fbbc05e3 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 50 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824102:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408cf1a00/0x1e260726fbbc05e3 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 50 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824110:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408cf1400/0x1e260726fbbc05ea lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 51 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824114:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408cf1400/0x1e260726fbbc05ea lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 51 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824122:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408cf0c00/0x1e260726fbbc05f1 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 52 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824128:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408cf0c00/0x1e260726fbbc05f1 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 52 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824142:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408cf1200/0x1e260726fbbc05f8 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 53 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824147:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408cf1200/0x1e260726fbbc05f8 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 53 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824162:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408cf0400/0x1e260726fbbc05ff lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 54 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824168:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408cf0400/0x1e260726fbbc05ff lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 54 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824189:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408cf0e00/0x1e260726fbbc0606 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 55 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824193:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408cf0e00/0x1e260726fbbc0606 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 55 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824559:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408cf1000/0x1e260726fbbc060d lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 56 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824565:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408cf1000/0x1e260726fbbc060d lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 56 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824589:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408cf0800/0x1e260726fbbc0614 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 57 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824593:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408cf0800/0x1e260726fbbc0614 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 57 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824602:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408cf0200/0x1e260726fbbc061b lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 58 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824606:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408cf0200/0x1e260726fbbc061b lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 58 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824613:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408cf1800/0x1e260726fbbc0622 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 59 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824617:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408cf1800/0x1e260726fbbc0622 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 59 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824624:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408cf1e00/0x1e260726fbbc0629 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 60 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824628:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408cf1e00/0x1e260726fbbc0629 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 60 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824635:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408cf1c00/0x1e260726fbbc0630 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 61 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824639:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408cf1c00/0x1e260726fbbc0630 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 61 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824647:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408cf0600/0x1e260726fbbc0637 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 62 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824651:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408cf0600/0x1e260726fbbc0637 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 62 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824665:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537800/0x1e260726fbbc063e lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 63 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824669:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537800/0x1e260726fbbc063e lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 63 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824680:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537e00/0x1e260726fbbc0645 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 64 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824683:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537e00/0x1e260726fbbc0645 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 64 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824693:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537c00/0x1e260726fbbc064c lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 65 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824702:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537c00/0x1e260726fbbc064c lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 65 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824711:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536600/0x1e260726fbbc0653 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 66 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824714:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536600/0x1e260726fbbc0653 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 66 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824722:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc065a lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 67 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824726:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc065a lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 67 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824804:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537600/0x1e260726fbbc0661 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 68 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824810:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537600/0x1e260726fbbc0661 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 68 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824923:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0668 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 69 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.824928:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0668 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 69 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.861749:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572041.978530:0:62539:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc06b5 lrc: 1/0,0 mode: --/PR res: [0x200009871:0x10:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4de5db expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572041.978539:0:62539:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572041.978542:0:62539:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc06b5 lrc: 0/0,0 mode: --/PR res: [0x200009871:0x10:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4de5db expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000020:00100000:2.0:1788572041.978563:0:62539:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572041.978582:0:62539:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c63100 x1875450488396416/t0(0) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572041.978601:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488396416:192.168.202.125@tcp:36:rm.0 00000100:00100000:2.0:1788572041.978760:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416ec5240 going to sleep for 11 seconds 00000100:00100000:2.0:1788572041.982049:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488396416:192.168.202.125@tcp:36:rm.0 00000080:00200000:2.0:1788572041.982095:0:62539:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200009871:0x10:0x0](ffff94d403684b08) 00000080:00200000:2.0:1788572041.982170:0:62539:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572041.982340:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572041.982357:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 175 lru_type: 1 00010000:00010000:2.0:1788572041.982371:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 175 lru_type: 1 00000080:00200000:2.0:1788572041.982379:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572041.982385:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572041.982389:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:2.0:1788572041.982396:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a41f8fb58 00010000:00010000:2.0:1788572041.982400:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 175 lru_type: 1 00010000:00010000:2.0:1788572041.982408:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 176 lru_type: 1 00000080:00200000:2.0:1788572041.982422:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572041.982428:0:62539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=24 00000080:00200000:2.0:1788572041.982433:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572041.982454:0:62539:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=dir-8, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), flags=8 00000080:00200000:2.0:1788572041.982458:0:62539:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=dir-8, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), intent=getattr 00000002:00010000:2.0:1788572041.982484:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: dir-8,[0x0:0x0:0x0]) in obj [0x200009871:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572041.982525:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537e00/0x1e260726fbbc06c3 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.982538:0:62539:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537e00/0x1e260726fbbc06c3 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.982548:0:62539:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537e00/0x1e260726fbbc06c3 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572041.982567:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488396544:192.168.202.125@tcp:101:rm.0 00000100:00100000:2.0:1788572041.982669:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416ec5240 going to sleep for 11 seconds 00000100:00100000:2.0:1788572041.995863:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488396544:192.168.202.125@tcp:101:rm.0 00010000:00010000:2.0:1788572041.995890:0:62539:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537e00/0x1e260726fbbc06c3 lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4de61a expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.995924:0:62539:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537e00/0x1e260726fbbc06c3 lrc: 4/1,0 mode: --/PR res: [0x200009871:0x12:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4de61a expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.995933:0:62539:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537e00/0x1e260726fbbc06c3 lrc: 4/1,0 mode: --/PR res: [0x200009871:0x12:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4de61a expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.995945:0:62539:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537e00/0x1e260726fbbc06c3 lrc: 4/1,0 mode: PR/PR res: [0x200009871:0x12:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4de61a expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.995957:0:62539:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537e00/0x1e260726fbbc06c3 lrc: 4/1,0 mode: PR/PR res: [0x200009871:0x12:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4de61a expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.995973:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537e00/0x1e260726fbbc06c3 lrc: 5/2,0 mode: PR/PR res: [0x200009871:0x12:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4de61a expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.995997:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537e00/0x1e260726fbbc06c3 lrc: 5/2,0 mode: PR/PR res: [0x200009871:0x12:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4de61a expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.996006:0:62539:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537e00/0x1e260726fbbc06c3 lrc: 4/1,0 mode: PR/PR res: [0x200009871:0x12:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4de61a expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572041.996021:0:62539:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408c63480 x1875450488396544/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/584 e 0 to 0 dl 1788572057 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572041.996036:0:62539:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537e00/0x1e260726fbbc06c3 lrc: 3/1,0 mode: PR/PR res: [0x200009871:0x12:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4de61a expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.996048:0:62539:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589973617/18 (0 0) 00000080:00200000:2.0:1788572041.996055:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200009871:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572041.996088:0:62539:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200009871:0x12:0x0], updating i_size 4096 00000080:00200000:2.0:1788572041.996091:0:62539:0:(namei.c:147:ll_iget()) got inode: [0x200009871:0x12:0x0](ffff94d403492a08): rc = 0 00000080:00010000:2.0:1788572041.996098:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200009871:0x12:0x0](ffff94d403492a08) for lock 0x1e260726fbbc06c3 00000080:00010000:2.0:1788572041.996119:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200009871:0x12:0x0](ffff94d403492a08) 00000080:00010000:2.0:1788572041.996123:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc06c3 from it ffffb68a41f8fb20 00010000:00010000:2.0:1788572041.996127:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537e00/0x1e260726fbbc06c3 lrc: 3/1,0 mode: PR/PR res: [0x200009871:0x12:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4de61a expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.996135:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537e00/0x1e260726fbbc06c3 lrc: 2/0,0 mode: PR/PR res: [0x200009871:0x12:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4de61a expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788572041.996174:0:62539:0:(file.c:6285:ll_getattr_dentry()) START file dir-8:[0x200009871:0x12:0x0](ffff94d403492a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572041.996191:0:62539:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200009871:0x12:0x0](ffff94d403492a08),name=dir-8 00000002:00010000:2.0:1788572041.996256:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009871:0x12:0x0]) in obj [0x200009871:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572041.996267:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537e00/0x1e260726fbbc06c3 lrc: 2/1,0 mode: PR/PR res: [0x200009871:0x12:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4de61a expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572041.996279:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537e00/0x1e260726fbbc06c3 lrc: 2/1,0 mode: PR/PR res: [0x200009871:0x12:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4de61a expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572041.996286:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200009871:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572041.996291:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200009871:0x12:0x0](ffff94d403492a08) 00000080:00010000:2.0:1788572041.996293:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200009871:0x12:0x0](ffff94d403492a08) for lock 0x1e260726fbbc06c3 00000080:00010000:2.0:1788572041.996297:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc06c3 from it ffffb68a41f8fcb8 00010000:00010000:2.0:1788572041.996300:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537e00/0x1e260726fbbc06c3 lrc: 3/1,0 mode: PR/PR res: [0x200009871:0x12:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4de61a expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572041.996308:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537e00/0x1e260726fbbc06c3 lrc: 2/0,0 mode: PR/PR res: [0x200009871:0x12:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4de61a expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572041.996372:0:62539:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572041.996381:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572041.996388:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 176 lru_type: 1 00010000:00010000:2.0:1788572041.996397:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 176 lru_type: 1 00000080:00200000:2.0:1788572041.996403:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572041.996407:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572041.996409:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:2.0:1788572041.996414:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a41f8fbf8 00010000:00010000:2.0:1788572041.996417:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 176 lru_type: 1 00010000:00010000:2.0:1788572041.996424:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 177 lru_type: 1 00000080:00200000:2.0:1788572041.996433:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572041.996439:0:62539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=26 00000080:00200000:2.0:1788572041.996443:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572041.996447:0:62539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=dir-8, flags=266 00000080:00200000:2.0:1788572041.996451:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x12:0x0](ffff94d403492a08), inode mode 41ed mask 44 00000080:00200008:2.0:1788572041.996460:0:62539:0:(file.c:1090:ll_file_open()) START file dir-8:[0x200009871:0x12:0x0](ffff94d403492a08), flags 704400 00000002:00010000:2.0:1788572041.996473:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009871:0x12:0x0]) in obj [0x200009871:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:2.0:1788572041.996513:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537800/0x1e260726fbbc06ca lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572041.996520:0:62539:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537800/0x1e260726fbbc06ca lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572041.996531:0:62539:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572041.996537:0:62539:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c63480 x1875450488396800/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572041.996546:0:62539:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537800/0x1e260726fbbc06ca lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572041.996565:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488396800:192.168.202.125@tcp:101:rm.0 00000100:00100000:2.0:1788572041.997011:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416ec5240 going to sleep for 11 seconds 00000100:00100000:2.0:1788572042.002404:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488396800:192.168.202.125@tcp:101:rm.0 00010000:00010000:2.0:1788572042.002426:0:62539:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537800/0x1e260726fbbc06ca lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572042.002440:0:62539:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537800/0x1e260726fbbc06ca lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572042.002446:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537800/0x1e260726fbbc06ca lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572042.002453:0:62539:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537800/0x1e260726fbbc06ca lrc: 3/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572042.002461:0:62539:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537800/0x1e260726fbbc06ca lrc: 4/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572042.002468:0:62539:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412537800 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:2.0:1788572042.002473:0:62539:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537800/0x1e260726fbbc06ca lrc: 4/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572042.002480:0:62539:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537800/0x1e260726fbbc06ca lrc: 4/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572042.002487:0:62539:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537800/0x1e260726fbbc06ca lrc: 5/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572042.002736:0:62539:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572042.002742:0:62539:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537800/0x1e260726fbbc06ca lrc: 3/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572042.002749:0:62539:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537800/0x1e260726fbbc06ca lrc: 0/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572042.002773:0:62539:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff94d408c63480 x1875450488396800/t85899346099(85899346099) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572057 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572042.002787:0:62539:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d408c63480 x1875450488396800/t85899346099(85899346099) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572057 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572042.002800:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200009871:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572042.002820:0:62539:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200009871:0x12:0x0], updating i_size 4096 00000002:00100000:2.0:1788572042.002841:0:62539:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d408c63480 x1875450488396800/t85899346099(85899346099) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572057 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788572042.002977:0:62539:0:(file.c:6285:ll_getattr_dentry()) START file dir-8:[0x200009871:0x12:0x0](ffff94d403492a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572042.002984:0:62539:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200009871:0x12:0x0](ffff94d403492a08),name=dir-8 00000002:00010000:2.0:1788572042.003018:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009871:0x12:0x0]) in obj [0x200009871:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572042.003033:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537e00/0x1e260726fbbc06c3 lrc: 2/1,0 mode: PR/PR res: [0x200009871:0x12:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4de61a expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572042.003045:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537e00/0x1e260726fbbc06c3 lrc: 2/1,0 mode: PR/PR res: [0x200009871:0x12:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4de61a expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572042.003052:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200009871:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572042.003058:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200009871:0x12:0x0](ffff94d403492a08) 00000080:00010000:2.0:1788572042.003061:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200009871:0x12:0x0](ffff94d403492a08) for lock 0x1e260726fbbc06c3 00000080:00010000:2.0:1788572042.003066:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc06c3 from it ffffb68a41f8fcd8 00010000:00010000:2.0:1788572042.003069:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537e00/0x1e260726fbbc06c3 lrc: 3/1,0 mode: PR/PR res: [0x200009871:0x12:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4de61a expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572042.003138:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537e00/0x1e260726fbbc06c3 lrc: 2/0,0 mode: PR/PR res: [0x200009871:0x12:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4de61a expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572042.003171:0:62539:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200009871:0x12:0x0](ffff94d403492a08) pos/size0/4096 32bit_api 0 00000002:00010000:2.0:1788572042.003193:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009871:0x12:0x0]) in obj [0x200009871:0x12:0x0], intent: readdir flags 0 00010000:00010000:2.0:1788572042.003201:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537e00/0x1e260726fbbc06c3 lrc: 2/1,0 mode: PR/PR res: [0x200009871:0x12:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4de61a expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572042.003816:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537e00/0x1e260726fbbc06c3 lrc: 2/1,0 mode: PR/PR res: [0x200009871:0x12:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4de61a expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1788572042.004430:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488396928:192.168.202.125@tcp:37:rm.0 00000100:00100000:2.0:1788572042.004512:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416ec5240 going to sleep for 11 seconds 00000100:00100000:2.0:1788572042.135018:0:42440:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875450511236352 00000100:00100000:2.0:1788572042.135029:0:42440:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875450511236352 00000100:00100000:2.0:1788572042.135100:0:42440:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.125@tcp, seq: 49 00000100:00100000:2.0:1788572042.135112:0:42440:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff94d408c57800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:9bf287c0-f582-4b83-84c1-15e766218620+5:26788:x1875450511236352:12345-192.168.202.125@tcp:104: 00010000:00010000:2.0:1788572042.135132:0:42440:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412166400/0x1e260726fbbc06ed lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0x2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xec901abfeb4de6b4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788572042.135260:0:42440:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788572042.135267:0:42440:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff94d408c57800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:9bf287c0-f582-4b83-84c1-15e766218620+5:26788:x1875450511236352:12345-192.168.202.125@tcp:104: Request processed in 158us (365us total) trans 0 rc 0/0 00000100:00100000:2.0:1788572042.135274:0:42440:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.125@tcp, seq: 49 00000100:00100000:2.0:1788572042.202953:0:2397:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.125@tcp (source: 12345-192.168.202.125@tcp) 00000100:00100000:2.0:1788572042.208382:0:2396:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.125@tcp (source: 12345-192.168.202.125@tcp) 00000100:00100000:2.0:1788572042.292520:0:42439:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875450511239296 00000100:00100000:2.0:1788572042.292529:0:42439:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875450511239296 00000100:00100000:2.0:1788572042.292859:0:42439:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.125@tcp, seq: 53 00000100:00100000:2.0:1788572042.292872:0:42439:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff94d408c54380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:9bf287c0-f582-4b83-84c1-15e766218620+5:23930:x1875450511239296:12345-192.168.202.125@tcp:104: 00010000:00010000:2.0:1788572042.292894:0:42439:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412166400/0x1e260726fbbc0725 lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xec901abfeb4de794 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788572042.292983:0:42439:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788572042.292990:0:42439:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff94d408c54380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:9bf287c0-f582-4b83-84c1-15e766218620+5:23930:x1875450511239296:12345-192.168.202.125@tcp:104: Request processed in 121us (991us total) trans 0 rc 0/0 00000100:00100000:2.0:1788572042.292997:0:42439:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.125@tcp, seq: 53 00000100:00100000:2.0:1788572042.367759:0:42440:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875450511240320 00000100:00100000:2.0:1788572042.367769:0:42440:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875450511240320 00000100:00100000:2.0:1788572042.367829:0:42440:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.125@tcp, seq: 54 00000100:00100000:2.0:1788572042.367849:0:42440:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff94d408c54e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:9bf287c0-f582-4b83-84c1-15e766218620+5:66199:x1875450511240320:12345-192.168.202.125@tcp:104: 00010000:00010000:2.0:1788572042.367867:0:42440:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc073a lrc: 2/0,0 mode: PR/PR res: [0x200009870:0x11:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xec901abfeb4de7f6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788572042.367946:0:42440:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788572042.367954:0:42440:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff94d408c54e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:9bf287c0-f582-4b83-84c1-15e766218620+5:66199:x1875450511240320:12345-192.168.202.125@tcp:104: Request processed in 107us (411us total) trans 0 rc 0/0 00000100:00100000:2.0:1788572042.367961:0:42440:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.125@tcp, seq: 54 00000100:00100000:2.0:1788572042.372784:0:2396:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.125@tcp (source: 12345-192.168.202.125@tcp) 00000100:00100000:2.0:1788572042.445038:0:42440:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875450511241600 00000100:00100000:2.0:1788572042.445047:0:42440:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875450511241600 00000100:00100000:2.0:1788572042.445105:0:42440:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.125@tcp, seq: 56 00000100:00100000:2.0:1788572042.445116:0:42440:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff94d408c57800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:9bf287c0-f582-4b83-84c1-15e766218620+5:66198:x1875450511241600:12345-192.168.202.125@tcp:104: 00010000:00010000:2.0:1788572042.445134:0:42440:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583800/0x1e260726fbbc0756 lrc: 2/0,0 mode: PR/PR res: [0x200009870:0x12:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xec901abfeb4de866 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788572042.445524:0:42440:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788572042.445533:0:42440:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff94d408c57800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:9bf287c0-f582-4b83-84c1-15e766218620+5:66198:x1875450511241600:12345-192.168.202.125@tcp:104: Request processed in 418us (574us total) trans 0 rc 0/0 00000100:00100000:2.0:1788572042.445540:0:42440:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.125@tcp, seq: 56 00000100:00100000:2.0:1788572042.450193:0:42439:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875450511241856 00000100:00100000:2.0:1788572042.450436:0:42439:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875450511241856 00000100:00100000:2.0:1788572042.450486:0:42439:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.125@tcp, seq: 57 00000100:00100000:2.0:1788572042.450495:0:42439:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff94d408c55500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:9bf287c0-f582-4b83-84c1-15e766218620+5:8443:x1875450511241856:12345-192.168.202.125@tcp:104: 00010000:00010000:2.0:1788572042.450512:0:42439:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc075d lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0x10:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xec901abfeb4de874 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788572042.450823:0:42439:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788572042.450832:0:42439:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff94d408c55500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:9bf287c0-f582-4b83-84c1-15e766218620+5:8443:x1875450511241856:12345-192.168.202.125@tcp:104: Request processed in 338us (762us total) trans 0 rc 0/0 00000100:00100000:2.0:1788572042.450839:0:42439:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.125@tcp, seq: 57 00000100:00100000:2.0:1788572042.511402:0:2396:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.125@tcp (source: 12345-192.168.202.125@tcp) 00000100:00100000:2.0:1788572042.516569:0:2396:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.125@tcp (source: 12345-192.168.202.125@tcp) 00000100:00100000:2.0:1788572042.589019:0:42439:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875450511244160 00000100:00100000:2.0:1788572042.589028:0:42439:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875450511244160 00000100:00100000:2.0:1788572042.589096:0:42439:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.125@tcp, seq: 60 00000100:00100000:2.0:1788572042.589104:0:42439:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff94d408c54380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:9bf287c0-f582-4b83-84c1-15e766218620+5:80443:x1875450511244160:12345-192.168.202.125@tcp:104: 00010000:00010000:2.0:1788572042.589120:0:42439:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc078e lrc: 2/0,0 mode: PR/PR res: [0x200009870:0x14:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xec901abfeb4de946 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788572042.589191:0:42439:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788572042.589196:0:42439:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff94d408c54380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:9bf287c0-f582-4b83-84c1-15e766218620+5:80443:x1875450511244160:12345-192.168.202.125@tcp:104: Request processed in 94us (286us total) trans 0 rc 0/0 00000100:00100000:2.0:1788572042.589770:0:42439:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.125@tcp, seq: 60 00000100:00100000:2.0:1788572042.594658:0:2397:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.125@tcp (source: 12345-192.168.202.125@tcp) 00000100:00100000:2.0:1788572042.682459:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488411008:192.168.202.125@tcp:37:rm.0 00000100:00100000:2.0:1788572042.682618:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416ec53c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572042.703807:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408d81840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572042.706353:0:2397:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.125@tcp (source: 12345-192.168.202.125@tcp) 00000100:00100000:2.0:1788572042.713075:0:42439:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875450511245696 00000100:00100000:2.0:1788572042.713084:0:42439:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875450511245696 00000100:00100000:2.0:1788572042.713140:0:42439:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.125@tcp, seq: 63 00000100:00100000:2.0:1788572042.713151:0:42439:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff94d408c57800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:9bf287c0-f582-4b83-84c1-15e766218620+5:8444:x1875450511245696:12345-192.168.202.125@tcp:104: 00010000:00010000:2.0:1788572042.713170:0:42439:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582600/0x1e260726fbbc07b1 lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0x13:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xec901abfeb4de9c4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788572042.713298:0:42439:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788572042.713306:0:42439:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff94d408c57800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:9bf287c0-f582-4b83-84c1-15e766218620+5:8444:x1875450511245696:12345-192.168.202.125@tcp:104: Request processed in 158us (636us total) trans 0 rc 0/0 00000100:00100000:2.0:1788572042.713313:0:42439:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.125@tcp, seq: 63 00000100:00100000:2.0:1788572042.779651:0:2396:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.125@tcp (source: 12345-192.168.202.125@tcp) 00000100:00100000:2.0:1788572042.785237:0:2396:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.125@tcp (source: 12345-192.168.202.125@tcp) 00000100:00100000:1.0:1788572042.852675:0:62539:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d404986300 x1875450488414592/t0(0) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572042.852698:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d404986300 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488414592:192.168.202.125@tcp:35:rm.0 00000100:00100000:1.0:1788572042.852890:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101000 going to sleep for 11 seconds 00000100:00100000:1.0:1788572042.855449:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d404986300 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488414592:192.168.202.125@tcp:35:rm.0 00000002:00100000:1.0:1788572042.855484:0:62539:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff94d404986680 x1875450488414080/t85899346132(85899346132) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 608/696 e 0 to 0 dl 1788572058 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572042.855562:0:62539:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572042.855682:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572042.855701:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 210 lru_type: 1 00010000:00010000:1.0:1788572042.855714:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 210 lru_type: 1 00000080:00200000:1.0:1788572042.855722:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572042.855729:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572042.855732:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:1.0:1788572042.855740:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a41f8fc10 00010000:00010000:1.0:1788572042.855743:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 210 lru_type: 1 00010000:00010000:1.0:1788572042.855751:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 211 lru_type: 1 00000080:00200000:1.0:1788572042.855765:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572042.855771:0:62539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=24 00000080:00200000:1.0:1788572042.855778:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788572042.855785:0:62539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=stripe_dir-17, flags=0 00000080:00200000:1.0:1788572042.855790:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 3 00000080:00200000:1.0:1788572042.855795:0:62539:0:(namei.c:2328:ll_rmdir()) VFS Op:name=stripe_dir-17, dir=[0x200009871:0x1:0x0](ffff94d4034900c8) 00010000:00010000:1.0:1788572042.856031:0:62539:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc07db lrc: 2/0,0 mode: PR/PR res: [0x200009871:0x25:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dea88 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572042.856086:0:62539:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc07db lrc: 1/0,0 mode: --/PR res: [0x200009871:0x25:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dea88 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572042.856092:0:62539:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572042.856096:0:62539:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc07db lrc: 0/0,0 mode: --/PR res: [0x200009871:0x25:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dea88 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:1.0:1788572042.856116:0:62539:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788572042.856124:0:62539:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d404986300 x1875450488414848/t0(0) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572042.856146:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d404986300 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488414848:192.168.202.125@tcp:36:rm.0 00000100:00100000:1.0:1788572042.856301:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101000 going to sleep for 11 seconds 00000100:00100000:2.0:1788572042.859519:0:2397:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.125@tcp (source: 12345-192.168.202.125@tcp) 00000100:00100000:1.0:1788572042.862119:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488414976:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572042.872703:0:2397:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.125@tcp (source: 12345-192.168.202.125@tcp) 00000100:00100000:1.0:1788572042.879009:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488414976:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572042.894909:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d404986300 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488414848:192.168.202.125@tcp:36:rm.0 00000080:00200000:1.0:1788572042.894968:0:62539:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200009871:0x25:0x0](ffff94d410bc3a88) 00000080:00200000:1.0:1788572042.895049:0:62539:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572042.895071:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572042.895088:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 211 lru_type: 1 00010000:00010000:1.0:1788572042.895104:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 211 lru_type: 1 00000080:00200000:1.0:1788572042.895112:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572042.895118:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572042.895122:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:1.0:1788572042.895130:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a41f8fb58 00010000:00010000:1.0:1788572042.895134:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 211 lru_type: 1 00010000:00010000:1.0:1788572042.895143:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 212 lru_type: 1 00000080:00200000:1.0:1788572042.895157:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572042.895163:0:62539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=24 00000080:00200000:1.0:1788572042.895169:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788572042.895419:0:62539:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=stripe_dir-18, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), flags=8 00000080:00200000:1.0:1788572042.895425:0:62539:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-18, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), intent=getattr 00000002:00010000:1.0:1788572042.895455:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: stripe_dir-18,[0x0:0x0:0x0]) in obj [0x200009871:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788572042.895501:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc07f7 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.895510:0:62539:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc07f7 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.895521:0:62539:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc07f7 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572042.895559:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d404986680 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488415232:192.168.202.125@tcp:101:rm.0 00000100:00100000:1.0:1788572042.895699:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101000 going to sleep for 11 seconds 00000100:00100000:1.0:1788572042.902508:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d404986680 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488415232:192.168.202.125@tcp:101:rm.0 00010000:00010000:1.0:1788572042.902664:0:62539:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc07f7 lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deaf8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.902691:0:62539:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc07f7 lrc: 4/1,0 mode: --/PR res: [0x200009871:0x27:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deaf8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.902699:0:62539:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc07f7 lrc: 4/1,0 mode: --/PR res: [0x200009871:0x27:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deaf8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.902709:0:62539:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc07f7 lrc: 4/1,0 mode: PR/PR res: [0x200009871:0x27:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deaf8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.902721:0:62539:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc07f7 lrc: 4/1,0 mode: PR/PR res: [0x200009871:0x27:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deaf8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.902735:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc07f7 lrc: 5/2,0 mode: PR/PR res: [0x200009871:0x27:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deaf8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.902744:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc07f7 lrc: 5/2,0 mode: PR/PR res: [0x200009871:0x27:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deaf8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.902751:0:62539:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc07f7 lrc: 4/1,0 mode: PR/PR res: [0x200009871:0x27:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deaf8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572042.902767:0:62539:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d404986680 x1875450488415232/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 584/672 e 0 to 0 dl 1788572058 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572042.902781:0:62539:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc07f7 lrc: 3/1,0 mode: PR/PR res: [0x200009871:0x27:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deaf8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.902792:0:62539:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589973617/39 (0 0) 00000080:00200000:1.0:1788572042.902799:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200009871:0x1:0x0] suppgids 0 -1: rc 0 00000002:00010000:1.0:1788572042.902925:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009870:0x18:0x0]) in obj [0x200009870:0x18:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572042.903062:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 3/1,0 mode: --/CR res: [0x200009870:0x18:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.903070:0:62539:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 3/1,0 mode: --/CR res: [0x200009870:0x18:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.903079:0:62539:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 3/1,0 mode: --/CR res: [0x200009870:0x18:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572042.903101:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d404985c00 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488415360:192.168.202.125@tcp:101:rm.0 00000100:00100000:1.0:1788572042.903284:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101000 going to sleep for 11 seconds 00000100:00100000:1.0:1788572042.909192:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d404985c00 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488415360:192.168.202.125@tcp:101:rm.0 00010000:00010000:1.0:1788572042.909355:0:62539:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 4/1,0 mode: --/CR res: [0x200009870:0x18:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb06 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.909368:0:62539:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 4/1,0 mode: --/PR res: [0x200009870:0x18:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb06 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.909379:0:62539:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 4/1,0 mode: PR/PR res: [0x200009870:0x18:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deb06 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.909392:0:62539:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 4/1,0 mode: PR/PR res: [0x200009870:0x18:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb06 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.909405:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 5/2,0 mode: PR/PR res: [0x200009870:0x18:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deb06 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.909413:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 5/2,0 mode: PR/PR res: [0x200009870:0x18:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deb06 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.909419:0:62539:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 4/1,0 mode: PR/PR res: [0x200009870:0x18:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deb06 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572042.909430:0:62539:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d404985c00 x1875450488415360/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/640 e 0 to 0 dl 1788572058 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572042.909444:0:62539:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 3/1,0 mode: PR/PR res: [0x200009870:0x18:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb06 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.909455:0:62539:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589973616/24 (0 0) 00010000:00010000:1.0:1788572042.909463:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 3/1,0 mode: PR/PR res: [0x200009870:0x18:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deb06 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.909472:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 2/0,0 mode: PR/PR res: [0x200009870:0x18:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb06 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:1.0:1788572042.909497:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2400090a2:0x16:0x0]) in obj [0x2400090a2:0x16:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572042.909549:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0805 lrc: 3/1,0 mode: --/CR res: [0x2400090a2:0x16:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.909557:0:62539:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0805 lrc: 3/1,0 mode: --/CR res: [0x2400090a2:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.909566:0:62539:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0805 lrc: 3/1,0 mode: --/CR res: [0x2400090a2:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572042.909692:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d404985c00 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488415488:192.168.202.125@tcp:101:rm.0 00000100:00100000:1.0:1788572042.909765:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101000 going to sleep for 11 seconds 00000100:00100000:1.0:1788572042.921697:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d404985c00 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488415488:192.168.202.125@tcp:101:rm.0 00010000:00010000:1.0:1788572042.921725:0:62539:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0805 lrc: 4/1,0 mode: --/CR res: [0x2400090a2:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb14 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.921737:0:62539:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0805 lrc: 4/1,0 mode: --/PR res: [0x2400090a2:0x16:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb14 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.921747:0:62539:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0805 lrc: 4/1,0 mode: PR/PR res: [0x2400090a2:0x16:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deb14 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.921757:0:62539:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0805 lrc: 4/1,0 mode: PR/PR res: [0x2400090a2:0x16:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb14 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.921770:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0805 lrc: 5/2,0 mode: PR/PR res: [0x2400090a2:0x16:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deb14 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.921779:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0805 lrc: 5/2,0 mode: PR/PR res: [0x2400090a2:0x16:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deb14 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.921791:0:62539:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0805 lrc: 4/1,0 mode: PR/PR res: [0x2400090a2:0x16:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deb14 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572042.921803:0:62539:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d404985c00 x1875450488415488/t0(0) o101->lustre-MDT0001-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/640 e 0 to 0 dl 1788572058 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572042.921817:0:62539:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0805 lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0x16:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb14 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.921830:0:62539:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d404a46c00 type 13 mode 4 res 9663713442/22 (0 0) 00010000:00010000:1.0:1788572042.921839:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0805 lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0x16:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deb14 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.921847:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0805 lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0x16:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb14 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572042.921881:0:62539:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200009871:0x27:0x0], updating i_size 8192 00000080:00200000:1.0:1788572042.921885:0:62539:0:(namei.c:147:ll_iget()) got inode: [0x200009871:0x27:0x0](ffff94d403539148): rc = 0 00000080:00010000:1.0:1788572042.921894:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200009871:0x27:0x0](ffff94d403539148) for lock 0x1e260726fbbc07f7 00000080:00010000:1.0:1788572042.921911:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200009871:0x27:0x0](ffff94d403539148) 00000080:00010000:1.0:1788572042.921914:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc07f7 from it ffffb68a41f8fb20 00010000:00010000:1.0:1788572042.921918:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc07f7 lrc: 3/1,0 mode: PR/PR res: [0x200009871:0x27:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deaf8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.921925:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc07f7 lrc: 2/0,0 mode: PR/PR res: [0x200009871:0x27:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deaf8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:1.0:1788572042.922064:0:62539:0:(file.c:6285:ll_getattr_dentry()) START file stripe_dir-18:[0x200009871:0x27:0x0](ffff94d403539148), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572042.922071:0:62539:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200009871:0x27:0x0](ffff94d403539148),name=stripe_dir-18 00000002:00010000:1.0:1788572042.922092:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009871:0x27:0x0]) in obj [0x200009871:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572042.922103:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc07f7 lrc: 2/1,0 mode: PR/PR res: [0x200009871:0x27:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deaf8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572042.922115:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc07f7 lrc: 2/1,0 mode: PR/PR res: [0x200009871:0x27:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deaf8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:1.0:1788572042.922124:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009870:0x18:0x0]) in obj [0x200009870:0x18:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572042.922130:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 2/1,0 mode: PR/PR res: [0x200009870:0x18:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb06 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572042.922138:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 2/1,0 mode: PR/PR res: [0x200009870:0x18:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb06 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572042.922147:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 3/1,0 mode: PR/PR res: [0x200009870:0x18:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deb06 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572042.922154:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 2/0,0 mode: PR/PR res: [0x200009870:0x18:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb06 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:1.0:1788572042.922163:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2400090a2:0x16:0x0]) in obj [0x2400090a2:0x16:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572042.922169:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0805 lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0x16:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb14 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572042.922176:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0805 lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0x16:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb14 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572042.922492:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0805 lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0x16:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deb14 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572042.922499:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0805 lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0x16:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb14 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572042.922510:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200009871:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1788572042.922516:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200009871:0x27:0x0](ffff94d403539148) 00000080:00010000:1.0:1788572042.922519:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200009871:0x27:0x0](ffff94d403539148) for lock 0x1e260726fbbc07f7 00000080:00010000:1.0:1788572042.922523:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc07f7 from it ffffb68a41f8fcb8 00010000:00010000:1.0:1788572042.922526:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc07f7 lrc: 3/1,0 mode: PR/PR res: [0x200009871:0x27:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deaf8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572042.922546:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc07f7 lrc: 2/0,0 mode: PR/PR res: [0x200009871:0x27:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deaf8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:1.0:1788572042.922559:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009870:0x18:0x0]) in obj [0x200009870:0x18:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572042.922750:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 2/1,0 mode: PR/PR res: [0x200009870:0x18:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb06 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572042.922763:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 2/1,0 mode: PR/PR res: [0x200009870:0x18:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb06 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572042.922772:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 3/1,0 mode: PR/PR res: [0x200009870:0x18:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deb06 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572042.922780:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 2/0,0 mode: PR/PR res: [0x200009870:0x18:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb06 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:1.0:1788572042.922791:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2400090a2:0x16:0x0]) in obj [0x2400090a2:0x16:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572042.922797:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0805 lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0x16:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb14 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572042.922804:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0805 lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0x16:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb14 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572042.922811:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0805 lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0x16:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deb14 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572042.922819:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0805 lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0x16:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb14 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788572042.922872:0:62539:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572042.922883:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572042.922890:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 212 lru_type: 1 00010000:00010000:1.0:1788572042.922899:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 212 lru_type: 1 00000080:00200000:1.0:1788572042.922904:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572042.922908:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572042.922911:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:1.0:1788572042.922915:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a41f8fbf8 00010000:00010000:1.0:1788572042.922918:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 212 lru_type: 1 00010000:00010000:1.0:1788572042.922926:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 213 lru_type: 1 00000080:00200000:1.0:1788572042.922933:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572042.922940:0:62539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=26 00000080:00200000:1.0:1788572042.922945:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788572042.923163:0:62539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=stripe_dir-18, flags=266 00000080:00200000:1.0:1788572042.923172:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x27:0x0](ffff94d403539148), inode mode 41ed mask 44 00000080:00200008:1.0:1788572042.923382:0:62539:0:(file.c:1090:ll_file_open()) START file stripe_dir-18:[0x200009871:0x27:0x0](ffff94d403539148), flags 704400 00000002:00010000:1.0:1788572042.923402:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009871:0x27:0x0]) in obj [0x200009871:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:1.0:1788572042.923451:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc080c lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.923459:0:62539:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc080c lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572042.923470:0:62539:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572042.923479:0:62539:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d404986680 x1875450488415744/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572042.923489:0:62539:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc080c lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572042.923509:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d404986680 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488415744:192.168.202.125@tcp:101:rm.0 00000100:00100000:1.0:1788572042.924040:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101000 going to sleep for 11 seconds 00000100:00100000:1.0:1788572042.929782:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d404986680 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488415744:192.168.202.125@tcp:101:rm.0 00010000:00010000:1.0:1788572042.929805:0:62539:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc080c lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.929830:0:62539:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc080c lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.929837:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc080c lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.929844:0:62539:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc080c lrc: 3/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.929852:0:62539:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc080c lrc: 4/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.929859:0:62539:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412582c00 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:1.0:1788572042.929863:0:62539:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc080c lrc: 4/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.929869:0:62539:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc080c lrc: 4/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.929876:0:62539:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc080c lrc: 5/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.929882:0:62539:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572042.929888:0:62539:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc080c lrc: 3/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572042.929895:0:62539:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc080c lrc: 0/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572042.929914:0:62539:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff94d404986680 x1875450488415744/t85899346135(85899346135) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/696 e 0 to 0 dl 1788572058 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572042.929930:0:62539:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d404986680 x1875450488415744/t85899346135(85899346135) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/696 e 0 to 0 dl 1788572058 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572042.929944:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200009871:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572042.929993:0:62539:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200009871:0x27:0x0], updating i_size 4096 00000002:00100000:1.0:1788572042.930010:0:62539:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d404986680 x1875450488415744/t85899346135(85899346135) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 608/696 e 0 to 0 dl 1788572058 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1788572042.930099:0:62539:0:(file.c:6285:ll_getattr_dentry()) START file stripe_dir-18:[0x200009871:0x27:0x0](ffff94d403539148), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572042.930106:0:62539:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200009871:0x27:0x0](ffff94d403539148),name=stripe_dir-18 00000002:00010000:1.0:1788572042.930126:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009871:0x27:0x0]) in obj [0x200009871:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572042.930142:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc07f7 lrc: 2/1,0 mode: PR/PR res: [0x200009871:0x27:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deaf8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572042.930155:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc07f7 lrc: 2/1,0 mode: PR/PR res: [0x200009871:0x27:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deaf8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:1.0:1788572042.930165:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009870:0x18:0x0]) in obj [0x200009870:0x18:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572042.930172:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 2/1,0 mode: PR/PR res: [0x200009870:0x18:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb06 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572042.931172:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 2/1,0 mode: PR/PR res: [0x200009870:0x18:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb06 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572042.931450:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 3/1,0 mode: PR/PR res: [0x200009870:0x18:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deb06 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572042.931460:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 2/0,0 mode: PR/PR res: [0x200009870:0x18:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb06 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00010000:1.0:1788572042.931481:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2400090a2:0x16:0x0]) in obj [0x2400090a2:0x16:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572042.931492:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0805 lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0x16:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb14 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572042.931502:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0805 lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0x16:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb14 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572042.931510:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0805 lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0x16:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deb14 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572042.931516:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0805 lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0x16:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb14 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788572042.931529:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200009871:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1788572042.931548:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200009871:0x27:0x0](ffff94d403539148) 00000080:00010000:1.0:1788572042.931552:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200009871:0x27:0x0](ffff94d403539148) for lock 0x1e260726fbbc07f7 00000080:00010000:1.0:1788572042.931557:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc07f7 from it ffffb68a41f8fcd8 00010000:00010000:1.0:1788572042.931561:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc07f7 lrc: 3/1,0 mode: PR/PR res: [0x200009871:0x27:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deaf8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572042.931752:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc07f7 lrc: 2/0,0 mode: PR/PR res: [0x200009871:0x27:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deaf8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:1.0:1788572042.931772:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009870:0x18:0x0]) in obj [0x200009870:0x18:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572042.931780:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 2/1,0 mode: PR/PR res: [0x200009870:0x18:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb06 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572042.931793:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 2/1,0 mode: PR/PR res: [0x200009870:0x18:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb06 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572042.931801:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 3/1,0 mode: PR/PR res: [0x200009870:0x18:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deb06 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572042.931906:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 2/0,0 mode: PR/PR res: [0x200009870:0x18:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb06 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000002:00010000:1.0:1788572042.931920:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2400090a2:0x16:0x0]) in obj [0x2400090a2:0x16:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572042.931926:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0805 lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0x16:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb14 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572042.931935:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0805 lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0x16:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb14 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572042.931942:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0805 lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0x16:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deb14 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572042.932122:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0805 lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0x16:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb14 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1788572042.932158:0:62539:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200009871:0x27:0x0](ffff94d403539148) pos/size0/8192 32bit_api 0 00010000:00010000:1.0:1788572042.932175:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582a00/0x1e260726fbbc0087 lrc: 2/0,0 mode: PR/PR res: [0x200009871:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd795 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00000002:00010000:1.0:1788572042.932369:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009870:0x18:0x0]) in obj [0x200009870:0x18:0x0], intent: readdir flags 0 00010000:00010000:1.0:1788572042.932377:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 2/1,0 mode: PR/PR res: [0x200009870:0x18:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb06 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572042.932386:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 2/1,0 mode: PR/PR res: [0x200009870:0x18:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb06 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000100:00100000:1.0:1788572042.936894:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d404985c00 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488415872:192.168.202.125@tcp:37:rm.0 00000100:00100000:1.0:1788572042.937022:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101000 going to sleep for 11 seconds 00000100:00100000:1.0:1788572042.955761:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d404985c00 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488415872:192.168.202.125@tcp:37:rm.0 00010000:00010000:1.0:1788572042.957082:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 3/1,0 mode: PR/PR res: [0x200009870:0x18:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deb06 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572042.957102:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc07fe lrc: 2/0,0 mode: PR/PR res: [0x200009870:0x18:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb06 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00010000:1.0:1788572042.957131:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2400090a2:0x16:0x0]) in obj [0x2400090a2:0x16:0x0], intent: readdir flags 0 00010000:00010000:1.0:1788572042.957147:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0805 lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0x16:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb14 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572042.957161:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0805 lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0x16:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb14 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000100:00100000:1.0:1788572042.962429:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d404985c00 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488416000:192.168.202.125@tcp:37:rm.0 00000100:00100000:1.0:1788572042.962699:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101900 going to sleep for 11 seconds 00000100:00100000:2.0:1788572043.000891:0:2396:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.125@tcp (source: 12345-192.168.202.125@tcp) 00000100:00100000:1.0:1788572043.002252:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488416640:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572043.018853:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488416640:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572043.027089:0:42440:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1788572043.027128:0:42440:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875450511249536 00000100:00100000:2.0:1788572043.027140:0:42440:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875450511249536 00000100:00100000:2.0:1788572043.027323:0:42440:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.125@tcp, seq: 69 00000100:00100000:2.0:1788572043.027337:0:42440:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff94d416f31050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:9bf287c0-f582-4b83-84c1-15e766218620+5:8444:x1875450511249536:12345-192.168.202.125@tcp:104: 00010000:00010000:2.0:1788572043.027362:0:42440:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0805 lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0x16:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xec901abfeb4deb14 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788572043.027459:0:42440:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788572043.027466:0:42440:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff94d416f31050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:9bf287c0-f582-4b83-84c1-15e766218620+5:8444:x1875450511249536:12345-192.168.202.125@tcp:104: Request processed in 133us (4056us total) trans 0 rc 0/0 00000100:00100000:2.0:1788572043.027474:0:42440:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.125@tcp, seq: 69 00000100:00100000:1.0:1788572043.028247:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488416768:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572043.047181:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488416768:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572043.106517:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488417664:192.168.202.125@tcp:37:rm.0 00000100:00100000:2.0:1788572043.106757:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416ec5c00 going to sleep for 11 seconds 00000100:00100000:1.0:1788572043.119444:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488417664:192.168.202.125@tcp:37:rm.0 00010000:00010000:1.0:1788572043.120680:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412167000/0x1e260726fbbc0821 lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0x17:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deb84 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572043.120700:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412167000/0x1e260726fbbc0821 lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0x17:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deb84 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1788572043.120765:0:62539:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200009871:0x29:0x0](ffff94d403538908) pos/size18446744073709551614/8192 32bit_api 0 00000080:00200008:1.0:1788572043.120783:0:62539:0:(file.c:479:ll_file_release()) START file stripe_dir-19:[0x200009871:0x29:0x0](ffff94d403538908), flags 704000 00000002:00100000:1.0:1788572043.121088:0:62539:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff94d408d3e300 x1875450488417408/t85899346138(85899346138) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 608/696 e 0 to 0 dl 1788572059 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572043.121100:0:62539:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572043.121106:0:62539:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d404986680 x1875450488417920/t0(0) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572043.121128:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d404986680 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488417920:192.168.202.125@tcp:35:rm.0 00000100:00100000:2.0:1788572043.121633:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101000 going to sleep for 11 seconds 00000100:00100000:2.0:1788572043.136122:0:2397:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.125@tcp (source: 12345-192.168.202.125@tcp) 00000100:00100000:2.0:1788572043.138516:0:42439:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.125@tcp, seq: 70 00000100:00100000:2.0:1788572043.138542:0:42439:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff94d408c60380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:9bf287c0-f582-4b83-84c1-15e766218620+5:80444:x1875450511250560:12345-192.168.202.125@tcp:104: 00010000:00010000:2.0:1788572043.138659:0:42439:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166800/0x1e260726fbbc081a lrc: 2/0,0 mode: PR/PR res: [0x200009870:0x19:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xec901abfeb4deb76 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788572043.138749:0:42439:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788572043.138756:0:42439:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff94d408c60380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:9bf287c0-f582-4b83-84c1-15e766218620+5:80444:x1875450511250560:12345-192.168.202.125@tcp:104: Request processed in 219us (2646us total) trans 0 rc 0/0 00000100:00100000:2.0:1788572043.138763:0:42439:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.125@tcp, seq: 70 00000100:00100000:1.0:1788572043.140350:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488418304:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572043.145626:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488418304:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572043.145763:0:42440:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875450511250816 00000100:00100000:2.0:1788572043.145773:0:42440:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875450511250816 00000100:00100000:2.0:1788572043.145837:0:42440:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.125@tcp, seq: 71 00000100:00100000:2.0:1788572043.145848:0:42440:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff94d408c55500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:9bf287c0-f582-4b83-84c1-15e766218620+5:8443:x1875450511250816:12345-192.168.202.125@tcp:104: 00010000:00010000:2.0:1788572043.145870:0:42440:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412167000/0x1e260726fbbc0821 lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0x17:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xec901abfeb4deb84 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788572043.145948:0:42440:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788572043.145956:0:42440:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff94d408c55500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:9bf287c0-f582-4b83-84c1-15e766218620+5:8443:x1875450511250816:12345-192.168.202.125@tcp:104: Request processed in 110us (464us total) trans 0 rc 0/0 00000100:00100000:2.0:1788572043.145963:0:42440:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.125@tcp, seq: 71 00000100:00100000:1.0:1788572043.146495:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488418432:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572043.161972:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488418432:192.168.202.125@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788572043.194772:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166800/0x1e260726fbbc0836 lrc: 2/1,0 mode: PR/PR res: [0x200009870:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debe6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572043.194783:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166800/0x1e260726fbbc0836 lrc: 3/1,0 mode: PR/PR res: [0x200009870:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4debe6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572043.194789:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166800/0x1e260726fbbc0836 lrc: 2/0,0 mode: PR/PR res: [0x200009870:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debe6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:2.0:1788572043.194809:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2400090a2:0x6:0x0]) in obj [0x2400090a2:0x6:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572043.194816:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc083d lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debf4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572043.194824:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc083d lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debf4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572043.194832:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc083d lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4debf4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572043.194839:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc083d lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debf4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572043.194854:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200009871:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1788572043.194860:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200009871:0x7:0x0](ffff94d410bcd348) 00000080:00010000:2.0:1788572043.194864:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200009871:0x7:0x0](ffff94d410bcd348) for lock 0x1e260726fbbc082f 00000080:00010000:2.0:1788572043.194867:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc082f from it ffffb68a41f8fcb8 00010000:00010000:2.0:1788572043.194871:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167000/0x1e260726fbbc082f lrc: 3/1,0 mode: PR/PR res: [0x200009871:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4debd8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572043.194878:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167000/0x1e260726fbbc082f lrc: 2/0,0 mode: PR/PR res: [0x200009871:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debd8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:2.0:1788572043.194889:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009870:0x8:0x0]) in obj [0x200009870:0x8:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572043.194895:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166800/0x1e260726fbbc0836 lrc: 2/1,0 mode: PR/PR res: [0x200009870:0x8:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debe6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572043.194903:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166800/0x1e260726fbbc0836 lrc: 2/1,0 mode: PR/PR res: [0x200009870:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debe6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572043.194912:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166800/0x1e260726fbbc0836 lrc: 3/1,0 mode: PR/PR res: [0x200009870:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4debe6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572043.194919:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166800/0x1e260726fbbc0836 lrc: 2/0,0 mode: PR/PR res: [0x200009870:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debe6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:2.0:1788572043.194926:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2400090a2:0x6:0x0]) in obj [0x2400090a2:0x6:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572043.194931:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc083d lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debf4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572043.194955:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc083d lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debf4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572043.194963:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc083d lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4debf4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572043.194970:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc083d lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debf4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572043.195022:0:62539:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572043.195032:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572043.195040:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 218 lru_type: 1 00010000:00010000:2.0:1788572043.195048:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 218 lru_type: 1 00000080:00200000:2.0:1788572043.195054:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572043.195058:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572043.195061:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:2.0:1788572043.195064:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a41f8fbf8 00010000:00010000:2.0:1788572043.195066:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 218 lru_type: 1 00010000:00010000:2.0:1788572043.195072:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 219 lru_type: 1 00000080:00200000:2.0:1788572043.195084:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572043.195090:0:62539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=26 00000080:00200000:2.0:1788572043.195094:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572043.195098:0:62539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=stripe_dir-2, flags=266 00000080:00200000:2.0:1788572043.195103:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x7:0x0](ffff94d410bcd348), inode mode 41ed mask 44 00000080:00200008:2.0:1788572043.195111:0:62539:0:(file.c:1090:ll_file_open()) START file stripe_dir-2:[0x200009871:0x7:0x0](ffff94d410bcd348), flags 704400 00000002:00010000:2.0:1788572043.195123:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009871:0x7:0x0]) in obj [0x200009871:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:2.0:1788572043.195168:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537600/0x1e260726fbbc0844 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572043.195175:0:62539:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537600/0x1e260726fbbc0844 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572043.195186:0:62539:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572043.195193:0:62539:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c63480 x1875450488419072/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572043.195222:0:62539:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537600/0x1e260726fbbc0844 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572043.195239:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488419072:192.168.202.125@tcp:101:rm.0 00000100:00100000:2.0:1788572043.195388:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416ec5b40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572043.199692:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488419072:192.168.202.125@tcp:101:rm.0 00010000:00010000:2.0:1788572043.199714:0:62539:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537600/0x1e260726fbbc0844 lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572043.199727:0:62539:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537600/0x1e260726fbbc0844 lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572043.199739:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537600/0x1e260726fbbc0844 lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572043.199745:0:62539:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537600/0x1e260726fbbc0844 lrc: 3/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572043.199754:0:62539:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537600/0x1e260726fbbc0844 lrc: 4/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572043.199760:0:62539:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412537600 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:2.0:1788572043.199765:0:62539:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537600/0x1e260726fbbc0844 lrc: 4/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572043.199772:0:62539:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537600/0x1e260726fbbc0844 lrc: 4/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572043.199778:0:62539:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537600/0x1e260726fbbc0844 lrc: 5/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572043.199785:0:62539:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572043.199792:0:62539:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537600/0x1e260726fbbc0844 lrc: 3/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572043.199856:0:62539:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537600/0x1e260726fbbc0844 lrc: 0/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572043.199878:0:62539:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff94d408c63480 x1875450488419072/t85899346141(85899346141) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/696 e 0 to 0 dl 1788572059 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572043.199891:0:62539:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d408c63480 x1875450488419072/t85899346141(85899346141) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/696 e 0 to 0 dl 1788572059 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572043.199907:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200009871:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572043.199948:0:62539:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200009871:0x7:0x0], updating i_size 4096 00000002:00100000:2.0:1788572043.199968:0:62539:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d408c63480 x1875450488419072/t85899346141(85899346141) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 608/696 e 0 to 0 dl 1788572059 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788572043.200042:0:62539:0:(file.c:6285:ll_getattr_dentry()) START file stripe_dir-2:[0x200009871:0x7:0x0](ffff94d410bcd348), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572043.200048:0:62539:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200009871:0x7:0x0](ffff94d410bcd348),name=stripe_dir-2 00000002:00010000:2.0:1788572043.200064:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009871:0x7:0x0]) in obj [0x200009871:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572043.200077:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167000/0x1e260726fbbc082f lrc: 2/1,0 mode: PR/PR res: [0x200009871:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debd8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572043.200089:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167000/0x1e260726fbbc082f lrc: 2/1,0 mode: PR/PR res: [0x200009871:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debd8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:2.0:1788572043.200098:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009870:0x8:0x0]) in obj [0x200009870:0x8:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572043.200103:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166800/0x1e260726fbbc0836 lrc: 2/1,0 mode: PR/PR res: [0x200009870:0x8:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debe6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572043.200110:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166800/0x1e260726fbbc0836 lrc: 2/1,0 mode: PR/PR res: [0x200009870:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debe6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572043.200118:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166800/0x1e260726fbbc0836 lrc: 3/1,0 mode: PR/PR res: [0x200009870:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4debe6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572043.200125:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166800/0x1e260726fbbc0836 lrc: 2/0,0 mode: PR/PR res: [0x200009870:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debe6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00010000:2.0:1788572043.200139:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2400090a2:0x6:0x0]) in obj [0x2400090a2:0x6:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572043.200147:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc083d lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debf4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572043.200155:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc083d lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debf4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572043.200163:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc083d lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4debf4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572043.200171:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc083d lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debf4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788572043.200190:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200009871:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1788572043.200195:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200009871:0x7:0x0](ffff94d410bcd348) 00000080:00010000:2.0:1788572043.200199:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200009871:0x7:0x0](ffff94d410bcd348) for lock 0x1e260726fbbc082f 00000080:00010000:2.0:1788572043.200244:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc082f from it ffffb68a41f8fcd8 00010000:00010000:2.0:1788572043.200248:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167000/0x1e260726fbbc082f lrc: 3/1,0 mode: PR/PR res: [0x200009871:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4debd8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572043.200256:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167000/0x1e260726fbbc082f lrc: 2/0,0 mode: PR/PR res: [0x200009871:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debd8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:2.0:1788572043.200267:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009870:0x8:0x0]) in obj [0x200009870:0x8:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572043.200273:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166800/0x1e260726fbbc0836 lrc: 2/1,0 mode: PR/PR res: [0x200009870:0x8:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debe6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572043.200281:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166800/0x1e260726fbbc0836 lrc: 2/1,0 mode: PR/PR res: [0x200009870:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debe6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572043.200295:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166800/0x1e260726fbbc0836 lrc: 3/1,0 mode: PR/PR res: [0x200009870:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4debe6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572043.200302:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166800/0x1e260726fbbc0836 lrc: 2/0,0 mode: PR/PR res: [0x200009870:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debe6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000002:00010000:2.0:1788572043.200308:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2400090a2:0x6:0x0]) in obj [0x2400090a2:0x6:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572043.200313:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc083d lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debf4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572043.200320:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc083d lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debf4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572043.200326:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc083d lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4debf4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572043.200333:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc083d lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debf4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1788572043.200789:0:62539:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200009871:0x7:0x0](ffff94d410bcd348) pos/size0/8192 32bit_api 0 00010000:00010000:2.0:1788572043.200861:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582a00/0x1e260726fbbc0087 lrc: 2/0,0 mode: PR/PR res: [0x200009871:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd795 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00000002:00010000:2.0:1788572043.200890:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009870:0x8:0x0]) in obj [0x200009870:0x8:0x0], intent: readdir flags 0 00010000:00010000:2.0:1788572043.200897:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166800/0x1e260726fbbc0836 lrc: 2/1,0 mode: PR/PR res: [0x200009870:0x8:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debe6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788572043.200910:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166800/0x1e260726fbbc0836 lrc: 2/1,0 mode: PR/PR res: [0x200009870:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debe6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000100:00100000:2.0:1788572043.201691:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488419200:192.168.202.125@tcp:37:rm.0 00000100:00100000:2.0:1788572043.202755:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416ec5b40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572043.210788:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488419200:192.168.202.125@tcp:37:rm.0 00010000:00010000:2.0:1788572043.211480:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166800/0x1e260726fbbc0836 lrc: 3/1,0 mode: PR/PR res: [0x200009870:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4debe6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788572043.211497:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166800/0x1e260726fbbc0836 lrc: 2/0,0 mode: PR/PR res: [0x200009870:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debe6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00010000:2.0:1788572043.211519:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2400090a2:0x6:0x0]) in obj [0x2400090a2:0x6:0x0], intent: readdir flags 0 00010000:00010000:2.0:1788572043.211534:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc083d lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debf4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788572043.211546:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc083d lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debf4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000100:00100000:2.0:1788572043.212142:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488419328:192.168.202.125@tcp:37:rm.0 00000100:00100000:2.0:1788572043.212254:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416ec5b40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572043.218865:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488419328:192.168.202.125@tcp:37:rm.0 00010000:00010000:2.0:1788572043.219736:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc083d lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4debf4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788572043.219758:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc083d lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4debf4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1788572043.219866:0:62539:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200009871:0x7:0x0](ffff94d410bcd348) pos/size18446744073709551614/8192 32bit_api 0 00000080:00200008:2.0:1788572043.219886:0:62539:0:(file.c:479:ll_file_release()) START file stripe_dir-2:[0x200009871:0x7:0x0](ffff94d410bcd348), flags 704000 00000002:00100000:2.0:1788572043.219963:0:62539:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff94d408c63480 x1875450488419072/t85899346141(85899346141) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 608/696 e 0 to 0 dl 1788572059 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572043.219974:0:62539:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572043.219980:0:62539:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c60a80 x1875450488419584/t0(0) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572043.220002:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488419584:192.168.202.125@tcp:35:rm.0 00000100:00100000:2.0:1788572043.220166:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416ec5b40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572043.227449:0:42440:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875450511251840 00000100:00100000:2.0:1788572043.227460:0:42440:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875450511251840 00000100:00100000:2.0:1788572043.227719:0:42440:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.125@tcp, seq: 72 00000100:00100000:2.0:1788572043.227731:0:42440:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff94d408c54380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:9bf287c0-f582-4b83-84c1-15e766218620+5:66973:x1875450511251840:12345-192.168.202.125@tcp:104: 00010000:00010000:2.0:1788572043.227749:0:42440:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166800/0x1e260726fbbc0836 lrc: 2/0,0 mode: PR/PR res: [0x200009870:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xec901abfeb4debe6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788572043.227840:0:42440:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788572043.227848:0:42440:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff94d408c54380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:9bf287c0-f582-4b83-84c1-15e766218620+5:66973:x1875450511251840:12345-192.168.202.125@tcp:104: Request processed in 120us (773us total) trans 0 rc 0/0 00000100:00100000:2.0:1788572043.227855:0:42440:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.125@tcp, seq: 72 00000100:00100000:2.0:1788572043.233993:0:2397:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.125@tcp (source: 12345-192.168.202.125@tcp) 00010000:00010000:1.0:1788572043.234356:0:42449:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788572043.234364:0:42449:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc083d lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xec901abfeb4debf4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572043.234373:0:42449:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412167600 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:1.0:1788572043.234378:0:42449:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc083d lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0x6:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xec901abfeb4debf4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572043.234388:0:42449:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc083d lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0x6:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xec901abfeb4debf4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572043.234396:0:42449:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc083d lrc: 3/0,0 mode: PR/PR res: [0x2400090a2:0x6:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xec901abfeb4debf4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572043.234537:0:42449:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc083d lrc: 2/0,0 mode: --/PR res: [0x2400090a2:0x6:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xec901abfeb4debf4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572043.234553:0:42449:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572043.234580:0:42449:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc083d lrc: 1/0,0 mode: --/PR res: [0x2400090a2:0x6:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xec901abfeb4debf4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572043.234588:0:42449:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc083d lrc: 0/0,0 mode: --/PR res: [0x2400090a2:0x6:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xec901abfeb4debf4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572043.234607:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572043.234609:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572043.234611:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572043.234625:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d404985c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488420096:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572043.246131:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d404985c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488420096:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572043.313957:0:2396:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.125@tcp (source: 12345-192.168.202.125@tcp) 00000100:00100000:1.0:1788572043.314609:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488421632:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572043.318679:0:42439:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875450511255040 00000100:00100000:2.0:1788572043.318687:0:42439:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875450511255040 00000100:00100000:1.0:1788572043.318740:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488421632:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572043.318784:0:42439:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.125@tcp, seq: 75 00000100:00100000:2.0:1788572043.318794:0:42439:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff94d408c54e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:9bf287c0-f582-4b83-84c1-15e766218620+5:26788:x1875450511255040:12345-192.168.202.125@tcp:104: 00010000:00010000:2.0:1788572043.318811:0:42439:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412166c00/0x1e260726fbbc0859 lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xec901abfeb4dec64 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788572043.318874:0:42439:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788572043.318881:0:42439:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff94d408c54e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:9bf287c0-f582-4b83-84c1-15e766218620+5:26788:x1875450511255040:12345-192.168.202.125@tcp:104: Request processed in 89us (307us total) trans 0 rc 0/0 00000100:00100000:2.0:1788572043.318888:0:42439:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.125@tcp, seq: 75 00010000:00010000:1.0:1788572043.318898:0:42449:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788572043.318904:0:42449:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412166c00/0x1e260726fbbc0859 lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xec901abfeb4dec64 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572043.318913:0:42449:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412166c00 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:1.0:1788572043.318917:0:42449:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412166c00/0x1e260726fbbc0859 lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0x7:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xec901abfeb4dec64 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572043.318925:0:42449:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412166c00/0x1e260726fbbc0859 lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0x7:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xec901abfeb4dec64 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572043.318945:0:42449:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412166c00/0x1e260726fbbc0859 lrc: 3/0,0 mode: PR/PR res: [0x2400090a2:0x7:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xec901abfeb4dec64 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572043.319045:0:42449:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412166c00/0x1e260726fbbc0859 lrc: 2/0,0 mode: --/PR res: [0x2400090a2:0x7:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xec901abfeb4dec64 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572043.319052:0:42449:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572043.319081:0:42449:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412166c00/0x1e260726fbbc0859 lrc: 1/0,0 mode: --/PR res: [0x2400090a2:0x7:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xec901abfeb4dec64 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572043.319089:0:42449:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412166c00/0x1e260726fbbc0859 lrc: 0/0,0 mode: --/PR res: [0x2400090a2:0x7:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xec901abfeb4dec64 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572043.319110:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572043.319112:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572043.319113:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572043.319119:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788572043.319130:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d404985c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488421760:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572043.324861:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d404985c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488421760:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572043.412070:0:42440:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1788572043.412088:0:42440:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875450511256064 00000100:00100000:2.0:1788572043.412095:0:42440:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875450511256064 00000100:00100000:2.0:1788572043.412149:0:42440:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.125@tcp, seq: 76 00000100:00100000:2.0:1788572043.412158:0:42440:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff94d41541c450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:9bf287c0-f582-4b83-84c1-15e766218620+5:66199:x1875450511256064:12345-192.168.202.125@tcp:104: 00010000:00010000:2.0:1788572043.412174:0:42440:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166800/0x1e260726fbbc086e lrc: 2/0,0 mode: PR/PR res: [0x200009870:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xec901abfeb4decc6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788572043.412280:0:42440:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788572043.412287:0:42440:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff94d41541c450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:9bf287c0-f582-4b83-84c1-15e766218620+5:66199:x1875450511256064:12345-192.168.202.125@tcp:104: Request processed in 130us (1289us total) trans 0 rc 0/0 00000100:00100000:2.0:1788572043.412295:0:42440:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.125@tcp, seq: 76 00000100:00100000:1.0:1788572043.412533:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488423296:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572043.417137:0:2397:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.125@tcp (source: 12345-192.168.202.125@tcp) 00000100:00100000:1.0:1788572043.417893:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488423296:192.168.202.125@tcp:103:ldlm_bl.0 00000080:00200000:1.0:1788572043.479621:0:62539:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200009871:0xd:0x0](ffff94d403491988) pos/size18446744073709551614/8192 32bit_api 0 00000080:00200008:1.0:1788572043.479640:0:62539:0:(file.c:479:ll_file_release()) START file stripe_dir-5:[0x200009871:0xd:0x0](ffff94d403491988), flags 704000 00000002:00100000:1.0:1788572043.479700:0:62539:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff94d408d3ed80 x1875450488424064/t85899346150(85899346150) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 608/696 e 0 to 0 dl 1788572059 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572043.479712:0:62539:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572043.479717:0:62539:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d404985c00 x1875450488424576/t0(0) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572043.479739:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d404985c00 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488424576:192.168.202.125@tcp:35:rm.0 00000100:00100000:1.0:1788572043.479834:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572043.483527:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d404985c00 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488424576:192.168.202.125@tcp:35:rm.0 00000002:00100000:1.0:1788572043.483557:0:62539:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff94d408d3ed80 x1875450488424064/t85899346150(85899346150) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 608/696 e 0 to 0 dl 1788572059 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572043.483615:0:62539:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572043.483634:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.483652:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 228 lru_type: 1 00010000:00010000:1.0:1788572043.483666:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 228 lru_type: 1 00000080:00200000:1.0:1788572043.483674:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572043.483680:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572043.483683:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:1.0:1788572043.483689:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a41f8fc10 00010000:00010000:1.0:1788572043.483692:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 228 lru_type: 1 00010000:00010000:1.0:1788572043.483699:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 229 lru_type: 1 00000080:00200000:1.0:1788572043.483710:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572043.483715:0:62539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=24 00000080:00200000:1.0:1788572043.483719:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788572043.483724:0:62539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=stripe_dir-5, flags=0 00000080:00200000:1.0:1788572043.483728:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 3 00000080:00200000:1.0:1788572043.483732:0:62539:0:(namei.c:2328:ll_rmdir()) VFS Op:name=stripe_dir-5, dir=[0x200009871:0x1:0x0](ffff94d4034900c8) 00010000:00010000:1.0:1788572043.483749:0:62539:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166a00/0x1e260726fbbc0883 lrc: 2/0,0 mode: PR/PR res: [0x200009871:0xd:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4ded28 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572043.483876:0:62539:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166a00/0x1e260726fbbc0883 lrc: 1/0,0 mode: --/PR res: [0x200009871:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4ded28 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572043.483885:0:62539:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572043.483887:0:62539:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166a00/0x1e260726fbbc0883 lrc: 0/0,0 mode: --/PR res: [0x200009871:0xd:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4ded28 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:1.0:1788572043.483906:0:62539:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788572043.483910:0:62539:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d404985c00 x1875450488424832/t0(0) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572043.483937:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d404985c00 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488424832:192.168.202.125@tcp:36:rm.0 00000100:00100000:1.0:1788572043.484154:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101a80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572043.486415:0:42440:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875450511257344 00000100:00100000:2.0:1788572043.486424:0:42440:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875450511257344 00000100:00100000:2.0:1788572043.486478:0:42440:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.125@tcp, seq: 78 00000100:00100000:2.0:1788572043.486488:0:42440:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff94d408c57b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:9bf287c0-f582-4b83-84c1-15e766218620+5:80443:x1875450511257344:12345-192.168.202.125@tcp:104: 00010000:00010000:2.0:1788572043.486505:0:42440:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166800/0x1e260726fbbc088a lrc: 2/0,0 mode: PR/PR res: [0x200009870:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xec901abfeb4ded36 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788572043.486623:0:42440:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788572043.486630:0:42440:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff94d408c57b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:9bf287c0-f582-4b83-84c1-15e766218620+5:80443:x1875450511257344:12345-192.168.202.125@tcp:104: Request processed in 144us (278us total) trans 0 rc 0/0 00000100:00100000:2.0:1788572043.486637:0:42440:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.125@tcp, seq: 78 00000100:00100000:2.0:1788572043.492913:0:42439:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875450511257600 00000100:00100000:2.0:1788572043.492922:0:42439:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875450511257600 00000100:00100000:2.0:1788572043.493046:0:42439:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.125@tcp, seq: 79 00000100:00100000:2.0:1788572043.493058:0:42439:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff94d408c55180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:9bf287c0-f582-4b83-84c1-15e766218620+5:23930:x1875450511257600:12345-192.168.202.125@tcp:104: 00010000:00010000:2.0:1788572043.493075:0:42439:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412167000/0x1e260726fbbc0891 lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xec901abfeb4ded44 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788572043.493190:0:42439:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788572043.493197:0:42439:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff94d408c55180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:9bf287c0-f582-4b83-84c1-15e766218620+5:23930:x1875450511257600:12345-192.168.202.125@tcp:104: Request processed in 143us (339us total) trans 0 rc 0/0 00000100:00100000:2.0:1788572043.493233:0:42439:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.125@tcp, seq: 79 00000100:00100000:1.0:1788572043.509272:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d404985c00 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488424832:192.168.202.125@tcp:36:rm.0 00000080:00200000:1.0:1788572043.509312:0:62539:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200009871:0xd:0x0](ffff94d403491988) 00000080:00200000:1.0:1788572043.509372:0:62539:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572043.509386:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.509398:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 229 lru_type: 1 00010000:00010000:1.0:1788572043.509408:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 229 lru_type: 1 00000080:00200000:1.0:1788572043.509413:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572043.509417:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572043.509419:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:1.0:1788572043.509425:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a41f8fb58 00010000:00010000:1.0:1788572043.509428:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 229 lru_type: 1 00010000:00010000:1.0:1788572043.509442:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 230 lru_type: 1 00000080:00200000:1.0:1788572043.509456:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572043.509463:0:62539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=24 00000080:00200000:1.0:1788572043.509467:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788572043.509479:0:62539:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=stripe_dir-6, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), flags=8 00000080:00200000:1.0:1788572043.509483:0:62539:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-6, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), intent=getattr 00000002:00010000:1.0:1788572043.509514:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: stripe_dir-6,[0x0:0x0:0x0]) in obj [0x200009871:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788572043.509558:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc089f lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.509566:0:62539:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc089f lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.509575:0:62539:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc089f lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572043.509595:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d404986680 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488425216:192.168.202.125@tcp:101:rm.0 00000100:00100000:1.0:1788572043.509663:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572043.514653:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d404986680 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488425216:192.168.202.125@tcp:101:rm.0 00010000:00010000:1.0:1788572043.514680:0:62539:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc089f lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4ded98 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.514701:0:62539:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc089f lrc: 4/1,0 mode: --/PR res: [0x200009871:0xf:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4ded98 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.514713:0:62539:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc089f lrc: 4/1,0 mode: --/PR res: [0x200009871:0xf:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4ded98 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.514723:0:62539:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc089f lrc: 4/1,0 mode: PR/PR res: [0x200009871:0xf:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4ded98 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.514732:0:62539:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc089f lrc: 4/1,0 mode: PR/PR res: [0x200009871:0xf:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4ded98 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.514745:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc089f lrc: 5/2,0 mode: PR/PR res: [0x200009871:0xf:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4ded98 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.514753:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc089f lrc: 5/2,0 mode: PR/PR res: [0x200009871:0xf:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4ded98 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.514761:0:62539:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc089f lrc: 4/1,0 mode: PR/PR res: [0x200009871:0xf:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4ded98 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572043.514772:0:62539:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d404986680 x1875450488425216/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 584/672 e 0 to 0 dl 1788572059 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572043.514785:0:62539:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc089f lrc: 3/1,0 mode: PR/PR res: [0x200009871:0xf:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4ded98 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.514807:0:62539:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589973617/15 (0 0) 00000080:00200000:1.0:1788572043.514814:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200009871:0x1:0x0] suppgids 0 -1: rc 0 00000002:00010000:1.0:1788572043.514873:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009870:0xc:0x0]) in obj [0x200009870:0xc:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.514921:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 3/1,0 mode: --/CR res: [0x200009870:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.514939:0:62539:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 3/1,0 mode: --/CR res: [0x200009870:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.514948:0:62539:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 3/1,0 mode: --/CR res: [0x200009870:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572043.514979:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488425344:192.168.202.125@tcp:101:rm.0 00000100:00100000:1.0:1788572043.515446:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572043.521577:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488425344:192.168.202.125@tcp:101:rm.0 00010000:00010000:1.0:1788572043.521607:0:62539:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 4/1,0 mode: --/CR res: [0x200009870:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deda6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.521616:0:62539:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 4/1,0 mode: --/PR res: [0x200009870:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deda6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.521623:0:62539:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 4/1,0 mode: PR/PR res: [0x200009870:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deda6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.521631:0:62539:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 4/1,0 mode: PR/PR res: [0x200009870:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deda6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.521641:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 5/2,0 mode: PR/PR res: [0x200009870:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deda6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.521646:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 5/2,0 mode: PR/PR res: [0x200009870:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deda6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.521658:0:62539:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 4/1,0 mode: PR/PR res: [0x200009870:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deda6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572043.521671:0:62539:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d4049bf480 x1875450488425344/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/640 e 0 to 0 dl 1788572059 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572043.521684:0:62539:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 3/1,0 mode: PR/PR res: [0x200009870:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deda6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.521697:0:62539:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589973616/12 (0 0) 00010000:00010000:1.0:1788572043.521705:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 3/1,0 mode: PR/PR res: [0x200009870:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deda6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.521713:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 2/0,0 mode: PR/PR res: [0x200009870:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deda6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:1.0:1788572043.521738:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2400090a2:0xa:0x0]) in obj [0x2400090a2:0xa:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.521777:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583800/0x1e260726fbbc08ad lrc: 3/1,0 mode: --/CR res: [0x2400090a2:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.521784:0:62539:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583800/0x1e260726fbbc08ad lrc: 3/1,0 mode: --/CR res: [0x2400090a2:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.521804:0:62539:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583800/0x1e260726fbbc08ad lrc: 3/1,0 mode: --/CR res: [0x2400090a2:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572043.521825:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488425472:192.168.202.125@tcp:101:rm.0 00000100:00100000:1.0:1788572043.521912:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572043.534036:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488425472:192.168.202.125@tcp:101:rm.0 00010000:00010000:1.0:1788572043.534064:0:62539:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583800/0x1e260726fbbc08ad lrc: 4/1,0 mode: --/CR res: [0x2400090a2:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dedb4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.534075:0:62539:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583800/0x1e260726fbbc08ad lrc: 4/1,0 mode: --/PR res: [0x2400090a2:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dedb4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.534087:0:62539:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583800/0x1e260726fbbc08ad lrc: 4/1,0 mode: PR/PR res: [0x2400090a2:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dedb4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.534097:0:62539:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583800/0x1e260726fbbc08ad lrc: 4/1,0 mode: PR/PR res: [0x2400090a2:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dedb4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.534111:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583800/0x1e260726fbbc08ad lrc: 5/2,0 mode: PR/PR res: [0x2400090a2:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dedb4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.534119:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583800/0x1e260726fbbc08ad lrc: 5/2,0 mode: PR/PR res: [0x2400090a2:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dedb4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.534126:0:62539:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583800/0x1e260726fbbc08ad lrc: 4/1,0 mode: PR/PR res: [0x2400090a2:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dedb4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572043.534138:0:62539:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d4049bf480 x1875450488425472/t0(0) o101->lustre-MDT0001-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/640 e 0 to 0 dl 1788572059 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572043.534150:0:62539:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583800/0x1e260726fbbc08ad lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dedb4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.534161:0:62539:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d404a46c00 type 13 mode 4 res 9663713442/10 (0 0) 00010000:00010000:1.0:1788572043.534170:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583800/0x1e260726fbbc08ad lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dedb4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.534349:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583800/0x1e260726fbbc08ad lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dedb4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572043.534387:0:62539:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200009871:0xf:0x0], updating i_size 8192 00000080:00200000:1.0:1788572043.534391:0:62539:0:(namei.c:147:ll_iget()) got inode: [0x200009871:0xf:0x0](ffff94d410bc1148): rc = 0 00000080:00010000:1.0:1788572043.534400:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200009871:0xf:0x0](ffff94d410bc1148) for lock 0x1e260726fbbc089f 00000080:00010000:1.0:1788572043.534418:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200009871:0xf:0x0](ffff94d410bc1148) 00000080:00010000:1.0:1788572043.534421:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc089f from it ffffb68a41f8fb20 00010000:00010000:1.0:1788572043.534425:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc089f lrc: 3/1,0 mode: PR/PR res: [0x200009871:0xf:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4ded98 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.534433:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc089f lrc: 2/0,0 mode: PR/PR res: [0x200009871:0xf:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4ded98 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:1.0:1788572043.534462:0:62539:0:(file.c:6285:ll_getattr_dentry()) START file stripe_dir-6:[0x200009871:0xf:0x0](ffff94d410bc1148), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572043.534470:0:62539:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200009871:0xf:0x0](ffff94d410bc1148),name=stripe_dir-6 00000002:00010000:1.0:1788572043.534487:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009871:0xf:0x0]) in obj [0x200009871:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.534497:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc089f lrc: 2/1,0 mode: PR/PR res: [0x200009871:0xf:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4ded98 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572043.534516:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc089f lrc: 2/1,0 mode: PR/PR res: [0x200009871:0xf:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4ded98 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:1.0:1788572043.534526:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009870:0xc:0x0]) in obj [0x200009870:0xc:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.534532:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 2/1,0 mode: PR/PR res: [0x200009870:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deda6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572043.534539:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 2/1,0 mode: PR/PR res: [0x200009870:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deda6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572043.534546:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 3/1,0 mode: PR/PR res: [0x200009870:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deda6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572043.534553:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 2/0,0 mode: PR/PR res: [0x200009870:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deda6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:1.0:1788572043.534560:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2400090a2:0xa:0x0]) in obj [0x2400090a2:0xa:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.534565:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583800/0x1e260726fbbc08ad lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0xa:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dedb4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572043.534572:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583800/0x1e260726fbbc08ad lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dedb4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572043.534579:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583800/0x1e260726fbbc08ad lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dedb4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572043.534587:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583800/0x1e260726fbbc08ad lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dedb4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572043.534595:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200009871:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1788572043.534600:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200009871:0xf:0x0](ffff94d410bc1148) 00000080:00010000:1.0:1788572043.534603:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200009871:0xf:0x0](ffff94d410bc1148) for lock 0x1e260726fbbc089f 00000080:00010000:1.0:1788572043.534607:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc089f from it ffffb68a41f8fcb8 00010000:00010000:1.0:1788572043.534610:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc089f lrc: 3/1,0 mode: PR/PR res: [0x200009871:0xf:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4ded98 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572043.534617:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc089f lrc: 2/0,0 mode: PR/PR res: [0x200009871:0xf:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4ded98 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:1.0:1788572043.534628:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009870:0xc:0x0]) in obj [0x200009870:0xc:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.534634:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 2/1,0 mode: PR/PR res: [0x200009870:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deda6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572043.534641:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 2/1,0 mode: PR/PR res: [0x200009870:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deda6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572043.534649:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 3/1,0 mode: PR/PR res: [0x200009870:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deda6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572043.534765:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 2/0,0 mode: PR/PR res: [0x200009870:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deda6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:1.0:1788572043.534834:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2400090a2:0xa:0x0]) in obj [0x2400090a2:0xa:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.534841:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583800/0x1e260726fbbc08ad lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0xa:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dedb4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572043.534848:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583800/0x1e260726fbbc08ad lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dedb4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572043.534855:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583800/0x1e260726fbbc08ad lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dedb4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572043.534861:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583800/0x1e260726fbbc08ad lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dedb4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788572043.534913:0:62539:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572043.534921:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.534939:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 230 lru_type: 1 00010000:00010000:1.0:1788572043.534946:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 230 lru_type: 1 00000080:00200000:1.0:1788572043.534952:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572043.534956:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572043.534959:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:1.0:1788572043.534963:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a41f8fbf8 00010000:00010000:1.0:1788572043.534966:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 230 lru_type: 1 00010000:00010000:1.0:1788572043.534973:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 231 lru_type: 1 00000080:00200000:1.0:1788572043.534981:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572043.534988:0:62539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=26 00000080:00200000:1.0:1788572043.534992:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788572043.534996:0:62539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=stripe_dir-6, flags=266 00000080:00200000:1.0:1788572043.535000:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0xf:0x0](ffff94d410bc1148), inode mode 41ed mask 44 00000080:00200008:1.0:1788572043.535008:0:62539:0:(file.c:1090:ll_file_open()) START file stripe_dir-6:[0x200009871:0xf:0x0](ffff94d410bc1148), flags 704400 00000002:00010000:1.0:1788572043.535021:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009871:0xf:0x0]) in obj [0x200009871:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:1.0:1788572043.535064:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08b4 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.535071:0:62539:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08b4 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572043.535082:0:62539:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572043.535091:0:62539:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d4049bf480 x1875450488425728/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572043.535099:0:62539:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08b4 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572043.535119:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488425728:192.168.202.125@tcp:101:rm.0 00000100:00100000:1.0:1788572043.535455:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572043.543115:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488425728:192.168.202.125@tcp:101:rm.0 00010000:00010000:1.0:1788572043.543134:0:62539:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08b4 lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.543145:0:62539:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08b4 lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.543150:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08b4 lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.543155:0:62539:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08b4 lrc: 3/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.543161:0:62539:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08b4 lrc: 4/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.543165:0:62539:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412582e00 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:1.0:1788572043.543168:0:62539:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08b4 lrc: 4/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.543173:0:62539:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08b4 lrc: 4/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.543178:0:62539:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08b4 lrc: 5/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.543193:0:62539:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572043.543200:0:62539:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08b4 lrc: 3/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.543251:0:62539:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08b4 lrc: 0/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572043.543272:0:62539:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff94d4049bf480 x1875450488425728/t85899346153(85899346153) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/696 e 0 to 0 dl 1788572059 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572043.543285:0:62539:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d4049bf480 x1875450488425728/t85899346153(85899346153) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/696 e 0 to 0 dl 1788572059 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572043.543303:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200009871:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572043.543338:0:62539:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200009871:0xf:0x0], updating i_size 4096 00000002:00100000:1.0:1788572043.543359:0:62539:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d4049bf480 x1875450488425728/t85899346153(85899346153) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 608/696 e 0 to 0 dl 1788572059 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1788572043.543428:0:62539:0:(file.c:6285:ll_getattr_dentry()) START file stripe_dir-6:[0x200009871:0xf:0x0](ffff94d410bc1148), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572043.543433:0:62539:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200009871:0xf:0x0](ffff94d410bc1148),name=stripe_dir-6 00000002:00010000:1.0:1788572043.543448:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009871:0xf:0x0]) in obj [0x200009871:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.543460:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc089f lrc: 2/1,0 mode: PR/PR res: [0x200009871:0xf:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4ded98 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572043.543473:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc089f lrc: 2/1,0 mode: PR/PR res: [0x200009871:0xf:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4ded98 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:1.0:1788572043.543479:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009870:0xc:0x0]) in obj [0x200009870:0xc:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.543482:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 2/1,0 mode: PR/PR res: [0x200009870:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deda6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572043.543487:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 2/1,0 mode: PR/PR res: [0x200009870:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deda6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572043.543492:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 3/1,0 mode: PR/PR res: [0x200009870:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deda6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572043.543497:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 2/0,0 mode: PR/PR res: [0x200009870:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deda6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00010000:1.0:1788572043.543508:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2400090a2:0xa:0x0]) in obj [0x2400090a2:0xa:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.543514:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583800/0x1e260726fbbc08ad lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0xa:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dedb4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572043.543521:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583800/0x1e260726fbbc08ad lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dedb4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572043.543527:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583800/0x1e260726fbbc08ad lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dedb4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572043.543534:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583800/0x1e260726fbbc08ad lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dedb4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788572043.543544:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200009871:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1788572043.543549:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200009871:0xf:0x0](ffff94d410bc1148) 00000080:00010000:1.0:1788572043.543553:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200009871:0xf:0x0](ffff94d410bc1148) for lock 0x1e260726fbbc089f 00000080:00010000:1.0:1788572043.543557:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc089f from it ffffb68a41f8fcd8 00010000:00010000:1.0:1788572043.543560:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc089f lrc: 3/1,0 mode: PR/PR res: [0x200009871:0xf:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4ded98 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572043.543567:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc089f lrc: 2/0,0 mode: PR/PR res: [0x200009871:0xf:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4ded98 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:1.0:1788572043.543575:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009870:0xc:0x0]) in obj [0x200009870:0xc:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.543877:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 2/1,0 mode: PR/PR res: [0x200009870:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deda6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572043.543885:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 2/1,0 mode: PR/PR res: [0x200009870:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deda6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572043.543891:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 3/1,0 mode: PR/PR res: [0x200009870:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deda6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572043.543896:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 2/0,0 mode: PR/PR res: [0x200009870:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deda6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000002:00010000:1.0:1788572043.543902:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2400090a2:0xa:0x0]) in obj [0x2400090a2:0xa:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.543906:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583800/0x1e260726fbbc08ad lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0xa:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dedb4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572043.543914:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583800/0x1e260726fbbc08ad lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dedb4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572043.543918:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583800/0x1e260726fbbc08ad lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dedb4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572043.543921:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583800/0x1e260726fbbc08ad lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dedb4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1788572043.546896:0:62539:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200009871:0xf:0x0](ffff94d410bc1148) pos/size0/8192 32bit_api 0 00010000:00010000:1.0:1788572043.546940:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582a00/0x1e260726fbbc0087 lrc: 2/0,0 mode: PR/PR res: [0x200009871:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd795 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00000002:00010000:1.0:1788572043.549418:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009870:0xc:0x0]) in obj [0x200009870:0xc:0x0], intent: readdir flags 0 00010000:00010000:1.0:1788572043.549443:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 2/1,0 mode: PR/PR res: [0x200009870:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deda6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572043.549460:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 2/1,0 mode: PR/PR res: [0x200009870:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deda6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000100:00100000:1.0:1788572043.551485:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488425856:192.168.202.125@tcp:37:rm.0 00000100:00100000:1.0:1788572043.551855:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572043.581455:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488425856:192.168.202.125@tcp:37:rm.0 00010000:00010000:1.0:1788572043.583147:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 3/1,0 mode: PR/PR res: [0x200009870:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deda6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572043.583174:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc08a6 lrc: 2/0,0 mode: PR/PR res: [0x200009870:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deda6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00010000:1.0:1788572043.583201:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2400090a2:0xa:0x0]) in obj [0x2400090a2:0xa:0x0], intent: readdir flags 0 00010000:00010000:1.0:1788572043.583246:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583800/0x1e260726fbbc08ad lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0xa:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dedb4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572043.583258:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583800/0x1e260726fbbc08ad lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dedb4 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000100:00100000:1.0:1788572043.611748:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488426624:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572043.620727:0:2396:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.125@tcp (source: 12345-192.168.202.125@tcp) 00000100:00100000:1.0:1788572043.623324:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488426624:192.168.202.125@tcp:103:ldlm_bl.0 00000002:00010000:1.0:1788572043.687778:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2400090a2:0xb:0x0]) in obj [0x2400090a2:0xb:0x0], intent: readdir flags 0 00010000:00010000:1.0:1788572043.687798:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412166c00/0x1e260726fbbc08c9 lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0xb:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee24 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572043.687812:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412166c00/0x1e260726fbbc08c9 lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee24 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000100:00100000:1.0:1788572043.688628:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488427648:192.168.202.125@tcp:37:rm.0 00000100:00100000:1.0:1788572043.688916:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572043.707089:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488427648:192.168.202.125@tcp:37:rm.0 00010000:00010000:1.0:1788572043.708256:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412166c00/0x1e260726fbbc08c9 lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dee24 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572043.708281:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412166c00/0x1e260726fbbc08c9 lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee24 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1788572043.708341:0:62539:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200009871:0x11:0x0](ffff94d4034963c8) pos/size18446744073709551614/8192 32bit_api 0 00000080:00200008:1.0:1788572043.708358:0:62539:0:(file.c:479:ll_file_release()) START file stripe_dir-7:[0x200009871:0x11:0x0](ffff94d4034963c8), flags 704000 00000002:00100000:1.0:1788572043.708427:0:62539:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff94d408d3ed80 x1875450488427392/t85899346156(85899346156) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 608/696 e 0 to 0 dl 1788572059 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572043.708438:0:62539:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572043.708443:0:62539:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d4049bf800 x1875450488427904/t0(0) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572043.708466:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488427904:192.168.202.125@tcp:35:rm.0 00000100:00100000:1.0:1788572043.708564:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572043.715098:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488427904:192.168.202.125@tcp:35:rm.0 00000002:00100000:1.0:1788572043.715137:0:62539:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff94d408d3ed80 x1875450488427392/t85899346156(85899346156) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 608/696 e 0 to 0 dl 1788572059 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572043.715496:0:62539:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572043.716441:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.716463:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 234 lru_type: 1 00010000:00010000:1.0:1788572043.716478:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 234 lru_type: 1 00000080:00200000:1.0:1788572043.716487:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572043.716496:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572043.716510:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:1.0:1788572043.716518:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a41f8fc10 00010000:00010000:1.0:1788572043.716523:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 234 lru_type: 1 00010000:00010000:1.0:1788572043.716532:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 235 lru_type: 1 00000080:00200000:1.0:1788572043.716548:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572043.716557:0:62539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=24 00000080:00200000:1.0:1788572043.716563:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788572043.716571:0:62539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=stripe_dir-7, flags=0 00000080:00200000:1.0:1788572043.716576:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 3 00000080:00200000:1.0:1788572043.716682:0:62539:0:(namei.c:2328:ll_rmdir()) VFS Op:name=stripe_dir-7, dir=[0x200009871:0x1:0x0](ffff94d4034900c8) 00010000:00010000:1.0:1788572043.716708:0:62539:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167000/0x1e260726fbbc08bb lrc: 2/0,0 mode: PR/PR res: [0x200009871:0x11:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dee08 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572043.716760:0:62539:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167000/0x1e260726fbbc08bb lrc: 1/0,0 mode: --/PR res: [0x200009871:0x11:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dee08 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572043.716768:0:62539:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572043.716771:0:62539:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167000/0x1e260726fbbc08bb lrc: 0/0,0 mode: --/PR res: [0x200009871:0x11:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dee08 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:1.0:1788572043.716866:0:62539:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788572043.716876:0:62539:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d4049bf800 x1875450488428160/t0(0) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572043.716900:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488428160:192.168.202.125@tcp:36:rm.0 00000100:00100000:1.0:1788572043.717108:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101a80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572043.723385:0:2396:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.125@tcp (source: 12345-192.168.202.125@tcp) 00000100:00100000:1.0:1788572043.724729:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488428288:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572043.731059:0:2396:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.125@tcp (source: 12345-192.168.202.125@tcp) 00000100:00100000:1.0:1788572043.731102:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488428288:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572043.756612:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4125d49c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572043.761912:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488428672:192.168.202.125@tcp:101:rm.0 00010000:00010000:1.0:1788572043.761958:0:62539:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030600/0x1e260726fbbc08de lrc: 4/1,0 mode: --/CR res: [0x200009870:0xe:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee86 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.761971:0:62539:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030600/0x1e260726fbbc08de lrc: 4/1,0 mode: --/PR res: [0x200009870:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee86 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.761983:0:62539:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030600/0x1e260726fbbc08de lrc: 4/1,0 mode: PR/PR res: [0x200009870:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dee86 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.761999:0:62539:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030600/0x1e260726fbbc08de lrc: 4/1,0 mode: PR/PR res: [0x200009870:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee86 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.762014:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030600/0x1e260726fbbc08de lrc: 5/2,0 mode: PR/PR res: [0x200009870:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dee86 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.762023:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030600/0x1e260726fbbc08de lrc: 5/2,0 mode: PR/PR res: [0x200009870:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dee86 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.762032:0:62539:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030600/0x1e260726fbbc08de lrc: 4/1,0 mode: PR/PR res: [0x200009870:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dee86 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572043.762052:0:62539:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408c55180 x1875450488428672/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/640 e 0 to 0 dl 1788572059 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572043.762070:0:62539:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030600/0x1e260726fbbc08de lrc: 3/1,0 mode: PR/PR res: [0x200009870:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee86 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.762083:0:62539:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589973616/14 (0 0) 00010000:00010000:1.0:1788572043.762095:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030600/0x1e260726fbbc08de lrc: 3/1,0 mode: PR/PR res: [0x200009870:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dee86 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.762104:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030600/0x1e260726fbbc08de lrc: 2/0,0 mode: PR/PR res: [0x200009870:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee86 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:1.0:1788572043.762131:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2400090a2:0xc:0x0]) in obj [0x2400090a2:0xc:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.762175:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 3/1,0 mode: --/CR res: [0x2400090a2:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.762183:0:62539:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 3/1,0 mode: --/CR res: [0x2400090a2:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.762192:0:62539:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 3/1,0 mode: --/CR res: [0x2400090a2:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572043.762258:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488428800:192.168.202.125@tcp:101:rm.0 00000100:00100000:1.0:1788572043.762336:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572043.772387:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488428800:192.168.202.125@tcp:101:rm.0 00010000:00010000:1.0:1788572043.772413:0:62539:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 4/1,0 mode: --/CR res: [0x2400090a2:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.772426:0:62539:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 4/1,0 mode: --/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.772439:0:62539:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 4/1,0 mode: PR/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.772451:0:62539:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 4/1,0 mode: PR/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.772465:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 5/2,0 mode: PR/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.772474:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 5/2,0 mode: PR/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.772483:0:62539:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 4/1,0 mode: PR/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572043.772495:0:62539:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d4049bd180 x1875450488428800/t0(0) o101->lustre-MDT0001-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/640 e 0 to 0 dl 1788572059 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572043.772509:0:62539:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.772522:0:62539:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d404a46c00 type 13 mode 4 res 9663713442/12 (0 0) 00010000:00010000:1.0:1788572043.772531:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.772540:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572043.772581:0:62539:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200009871:0x13:0x0], updating i_size 8192 00000080:00200000:1.0:1788572043.772585:0:62539:0:(namei.c:147:ll_iget()) got inode: [0x200009871:0x13:0x0](ffff94d410bc1988): rc = 0 00000080:00010000:1.0:1788572043.772594:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200009871:0x13:0x0](ffff94d410bc1988) for lock 0x1e260726fbbc08d7 00000080:00010000:1.0:1788572043.772613:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200009871:0x13:0x0](ffff94d410bc1988) 00000080:00010000:1.0:1788572043.772617:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc08d7 from it ffffb68a41f8fb20 00010000:00010000:1.0:1788572043.772621:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc08d7 lrc: 3/1,0 mode: PR/PR res: [0x200009871:0x13:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dee78 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.772630:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc08d7 lrc: 2/0,0 mode: PR/PR res: [0x200009871:0x13:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee78 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:1.0:1788572043.772739:0:62539:0:(file.c:6285:ll_getattr_dentry()) START file stripe_dir-8:[0x200009871:0x13:0x0](ffff94d410bc1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572043.772746:0:62539:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200009871:0x13:0x0](ffff94d410bc1988),name=stripe_dir-8 00000002:00010000:1.0:1788572043.772766:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009871:0x13:0x0]) in obj [0x200009871:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.772777:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc08d7 lrc: 2/1,0 mode: PR/PR res: [0x200009871:0x13:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee78 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572043.772787:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc08d7 lrc: 2/1,0 mode: PR/PR res: [0x200009871:0x13:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee78 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:1.0:1788572043.772795:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009870:0xe:0x0]) in obj [0x200009870:0xe:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.772802:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030600/0x1e260726fbbc08de lrc: 2/1,0 mode: PR/PR res: [0x200009870:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee86 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572043.772809:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030600/0x1e260726fbbc08de lrc: 2/1,0 mode: PR/PR res: [0x200009870:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee86 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572043.772822:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030600/0x1e260726fbbc08de lrc: 3/1,0 mode: PR/PR res: [0x200009870:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dee86 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572043.772830:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030600/0x1e260726fbbc08de lrc: 2/0,0 mode: PR/PR res: [0x200009870:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee86 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:1.0:1788572043.772838:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2400090a2:0xc:0x0]) in obj [0x2400090a2:0xc:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.772844:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572043.772852:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572043.772859:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572043.772866:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572043.772876:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200009871:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1788572043.772881:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200009871:0x13:0x0](ffff94d410bc1988) 00000080:00010000:1.0:1788572043.772884:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200009871:0x13:0x0](ffff94d410bc1988) for lock 0x1e260726fbbc08d7 00000080:00010000:1.0:1788572043.772887:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc08d7 from it ffffb68a41f8fcb8 00010000:00010000:1.0:1788572043.772891:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc08d7 lrc: 3/1,0 mode: PR/PR res: [0x200009871:0x13:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dee78 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572043.772898:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc08d7 lrc: 2/0,0 mode: PR/PR res: [0x200009871:0x13:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee78 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:1.0:1788572043.772908:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009870:0xe:0x0]) in obj [0x200009870:0xe:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.772914:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030600/0x1e260726fbbc08de lrc: 2/1,0 mode: PR/PR res: [0x200009870:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee86 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572043.772935:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030600/0x1e260726fbbc08de lrc: 2/1,0 mode: PR/PR res: [0x200009870:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee86 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572043.772943:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030600/0x1e260726fbbc08de lrc: 3/1,0 mode: PR/PR res: [0x200009870:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dee86 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572043.772950:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030600/0x1e260726fbbc08de lrc: 2/0,0 mode: PR/PR res: [0x200009870:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee86 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:1.0:1788572043.772963:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2400090a2:0xc:0x0]) in obj [0x2400090a2:0xc:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.772968:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572043.772976:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572043.772984:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572043.772991:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788572043.773102:0:62539:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572043.773112:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.773121:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 236 lru_type: 1 00010000:00010000:1.0:1788572043.773187:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 236 lru_type: 1 00000080:00200000:1.0:1788572043.773195:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572043.773199:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572043.773232:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:1.0:1788572043.773237:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a41f8fbf8 00010000:00010000:1.0:1788572043.773240:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 236 lru_type: 1 00010000:00010000:1.0:1788572043.773249:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 237 lru_type: 1 00000080:00200000:1.0:1788572043.773258:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572043.773264:0:62539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=26 00000080:00200000:1.0:1788572043.773268:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788572043.773272:0:62539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=stripe_dir-8, flags=266 00000080:00200000:1.0:1788572043.773277:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x13:0x0](ffff94d410bc1988), inode mode 41ed mask 44 00000080:00200008:1.0:1788572043.773424:0:62539:0:(file.c:1090:ll_file_open()) START file stripe_dir-8:[0x200009871:0x13:0x0](ffff94d410bc1988), flags 704400 00000002:00010000:1.0:1788572043.773441:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009871:0x13:0x0]) in obj [0x200009871:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:1.0:1788572043.773482:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08ec lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.773490:0:62539:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08ec lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572043.773506:0:62539:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572043.773514:0:62539:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d4049bd180 x1875450488429056/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572043.773523:0:62539:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08ec lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572043.773542:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488429056:192.168.202.125@tcp:101:rm.0 00000100:00100000:1.0:1788572043.773845:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572043.797793:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488429056:192.168.202.125@tcp:101:rm.0 00010000:00010000:1.0:1788572043.797826:0:62539:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08ec lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.797841:0:62539:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08ec lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.797849:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08ec lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.797856:0:62539:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08ec lrc: 3/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.797865:0:62539:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08ec lrc: 4/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.797872:0:62539:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412583400 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:1.0:1788572043.797877:0:62539:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08ec lrc: 4/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.797888:0:62539:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08ec lrc: 4/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.797896:0:62539:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08ec lrc: 5/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.797903:0:62539:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572043.797909:0:62539:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08ec lrc: 3/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.797926:0:62539:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08ec lrc: 0/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572043.797955:0:62539:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff94d4049bd180 x1875450488429056/t85899346159(85899346159) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/696 e 0 to 0 dl 1788572059 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572043.797969:0:62539:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d4049bd180 x1875450488429056/t85899346159(85899346159) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/696 e 0 to 0 dl 1788572059 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572043.797996:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200009871:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572043.798028:0:62539:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200009871:0x13:0x0], updating i_size 4096 00000002:00100000:1.0:1788572043.798047:0:62539:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d4049bd180 x1875450488429056/t85899346159(85899346159) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 608/696 e 0 to 0 dl 1788572059 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1788572043.798122:0:62539:0:(file.c:6285:ll_getattr_dentry()) START file stripe_dir-8:[0x200009871:0x13:0x0](ffff94d410bc1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572043.798129:0:62539:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200009871:0x13:0x0](ffff94d410bc1988),name=stripe_dir-8 00000002:00010000:1.0:1788572043.798146:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009871:0x13:0x0]) in obj [0x200009871:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.798160:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc08d7 lrc: 2/1,0 mode: PR/PR res: [0x200009871:0x13:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee78 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572043.798177:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc08d7 lrc: 2/1,0 mode: PR/PR res: [0x200009871:0x13:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee78 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:1.0:1788572043.798187:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009870:0xe:0x0]) in obj [0x200009870:0xe:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.798194:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030600/0x1e260726fbbc08de lrc: 2/1,0 mode: PR/PR res: [0x200009870:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee86 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572043.798248:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030600/0x1e260726fbbc08de lrc: 2/1,0 mode: PR/PR res: [0x200009870:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee86 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572043.798259:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030600/0x1e260726fbbc08de lrc: 3/1,0 mode: PR/PR res: [0x200009870:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dee86 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572043.798267:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030600/0x1e260726fbbc08de lrc: 2/0,0 mode: PR/PR res: [0x200009870:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee86 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00010000:1.0:1788572043.798283:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2400090a2:0xc:0x0]) in obj [0x2400090a2:0xc:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.798291:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572043.798299:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572043.798308:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572043.798315:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788572043.798326:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200009871:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1788572043.798332:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200009871:0x13:0x0](ffff94d410bc1988) 00000080:00010000:1.0:1788572043.798335:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200009871:0x13:0x0](ffff94d410bc1988) for lock 0x1e260726fbbc08d7 00000080:00010000:1.0:1788572043.798340:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc08d7 from it ffffb68a41f8fcd8 00010000:00010000:1.0:1788572043.798343:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc08d7 lrc: 3/1,0 mode: PR/PR res: [0x200009871:0x13:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dee78 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572043.798351:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc08d7 lrc: 2/0,0 mode: PR/PR res: [0x200009871:0x13:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee78 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:1.0:1788572043.798362:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009870:0xe:0x0]) in obj [0x200009870:0xe:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.798368:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030600/0x1e260726fbbc08de lrc: 2/1,0 mode: PR/PR res: [0x200009870:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee86 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572043.798375:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030600/0x1e260726fbbc08de lrc: 2/1,0 mode: PR/PR res: [0x200009870:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee86 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572043.798383:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030600/0x1e260726fbbc08de lrc: 3/1,0 mode: PR/PR res: [0x200009870:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dee86 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572043.798391:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030600/0x1e260726fbbc08de lrc: 2/0,0 mode: PR/PR res: [0x200009870:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee86 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000002:00010000:1.0:1788572043.798398:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2400090a2:0xc:0x0]) in obj [0x2400090a2:0xc:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.798403:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572043.798415:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572043.798423:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572043.798430:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1788572043.798455:0:62539:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200009871:0x13:0x0](ffff94d410bc1988) pos/size0/8192 32bit_api 0 00010000:00010000:1.0:1788572043.798466:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582a00/0x1e260726fbbc0087 lrc: 2/0,0 mode: PR/PR res: [0x200009871:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd795 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00000002:00010000:1.0:1788572043.798493:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009870:0xe:0x0]) in obj [0x200009870:0xe:0x0], intent: readdir flags 0 00010000:00010000:1.0:1788572043.798500:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030600/0x1e260726fbbc08de lrc: 2/1,0 mode: PR/PR res: [0x200009870:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee86 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572043.798514:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030600/0x1e260726fbbc08de lrc: 2/1,0 mode: PR/PR res: [0x200009870:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee86 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000100:00100000:1.0:1788572043.799151:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488429184:192.168.202.125@tcp:37:rm.0 00000100:00100000:1.0:1788572043.799313:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572043.805848:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488429184:192.168.202.125@tcp:37:rm.0 00010000:00010000:1.0:1788572043.806503:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030600/0x1e260726fbbc08de lrc: 3/1,0 mode: PR/PR res: [0x200009870:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dee86 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572043.806526:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030600/0x1e260726fbbc08de lrc: 2/0,0 mode: PR/PR res: [0x200009870:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee86 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00010000:1.0:1788572043.806552:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2400090a2:0xc:0x0]) in obj [0x2400090a2:0xc:0x0], intent: readdir flags 0 00010000:00010000:1.0:1788572043.806568:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572043.806591:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000100:00100000:1.0:1788572043.807302:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488429312:192.168.202.125@tcp:37:rm.0 00000100:00100000:1.0:1788572043.807396:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572043.817328:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488429312:192.168.202.125@tcp:37:rm.0 00010000:00010000:1.0:1788572043.817845:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572043.817857:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1788572043.817903:0:62539:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200009871:0x13:0x0](ffff94d410bc1988) pos/size18446744073709551614/8192 32bit_api 0 00000080:00200008:1.0:1788572043.817927:0:62539:0:(file.c:479:ll_file_release()) START file stripe_dir-8:[0x200009871:0x13:0x0](ffff94d410bc1988), flags 704000 00000002:00100000:1.0:1788572043.817999:0:62539:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff94d4049bd180 x1875450488429056/t85899346159(85899346159) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 608/696 e 0 to 0 dl 1788572059 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572043.818010:0:62539:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572043.818015:0:62539:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d4049bc000 x1875450488429568/t0(0) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572043.818037:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488429568:192.168.202.125@tcp:35:rm.0 00000100:00100000:1.0:1788572043.818384:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572043.829059:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488429568:192.168.202.125@tcp:35:rm.0 00000002:00100000:1.0:1788572043.829091:0:62539:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff94d4049bd180 x1875450488429056/t85899346159(85899346159) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 608/696 e 0 to 0 dl 1788572059 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572043.829145:0:62539:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572043.829159:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.829173:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 237 lru_type: 1 00010000:00010000:1.0:1788572043.829183:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 237 lru_type: 1 00000080:00200000:1.0:1788572043.829189:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572043.829193:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572043.829195:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:1.0:1788572043.829233:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a41f8fc10 00010000:00010000:1.0:1788572043.829236:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 237 lru_type: 1 00010000:00010000:1.0:1788572043.829243:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 238 lru_type: 1 00000080:00200000:1.0:1788572043.829254:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572043.829259:0:62539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=24 00000080:00200000:1.0:1788572043.829264:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788572043.829269:0:62539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=stripe_dir-8, flags=0 00000080:00200000:1.0:1788572043.829273:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 3 00000080:00200000:1.0:1788572043.829277:0:62539:0:(namei.c:2328:ll_rmdir()) VFS Op:name=stripe_dir-8, dir=[0x200009871:0x1:0x0](ffff94d4034900c8) 00010000:00010000:1.0:1788572043.829295:0:62539:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc08d7 lrc: 2/0,0 mode: PR/PR res: [0x200009871:0x13:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dee78 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572043.829331:0:62539:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc08d7 lrc: 1/0,0 mode: --/PR res: [0x200009871:0x13:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dee78 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572043.829336:0:62539:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572043.829337:0:62539:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc08d7 lrc: 0/0,0 mode: --/PR res: [0x200009871:0x13:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dee78 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:1.0:1788572043.829354:0:62539:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788572043.829359:0:62539:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d4049bc000 x1875450488429824/t0(0) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572043.829374:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488429824:192.168.202.125@tcp:36:rm.0 00000100:00100000:1.0:1788572043.829433:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101a80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572043.841433:0:42440:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875450511261184 00000100:00100000:2.0:1788572043.841447:0:42440:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875450511261184 00000100:00100000:2.0:1788572043.841511:0:42440:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.125@tcp, seq: 84 00000100:00100000:2.0:1788572043.841522:0:42440:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff94d408c56680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:9bf287c0-f582-4b83-84c1-15e766218620+5:80443:x1875450511261184:12345-192.168.202.125@tcp:104: 00010000:00010000:2.0:1788572043.841539:0:42440:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030600/0x1e260726fbbc08de lrc: 2/0,0 mode: PR/PR res: [0x200009870:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xec901abfeb4dee86 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788572043.841628:0:42440:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788572043.841635:0:42440:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff94d408c56680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:9bf287c0-f582-4b83-84c1-15e766218620+5:80443:x1875450511261184:12345-192.168.202.125@tcp:104: Request processed in 115us (1270us total) trans 0 rc 0/0 00000100:00100000:2.0:1788572043.841642:0:42440:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.125@tcp, seq: 84 00000100:00100000:2.0:1788572043.855690:0:2396:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.125@tcp (source: 12345-192.168.202.125@tcp) 00010000:00010000:1.0:1788572043.856188:0:42449:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788572043.856195:0:42449:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572043.856233:0:42449:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412582e00 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:1.0:1788572043.856238:0:42449:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572043.856247:0:42449:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572043.856255:0:42449:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 3/0,0 mode: PR/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572043.856364:0:42449:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 2/0,0 mode: --/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572043.856371:0:42449:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572043.856388:0:42449:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 1/0,0 mode: --/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572043.856395:0:42449:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08e5 lrc: 0/0,0 mode: --/PR res: [0x2400090a2:0xc:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xec901abfeb4dee94 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572043.856410:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572043.856411:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572043.856413:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572043.856420:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788572043.856433:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488430080:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572043.863630:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488430080:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572043.876815:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488429824:192.168.202.125@tcp:36:rm.0 00000080:00200000:1.0:1788572043.876860:0:62539:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200009871:0x13:0x0](ffff94d410bc1988) 00000080:00200000:1.0:1788572043.876950:0:62539:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572043.876979:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.876995:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 238 lru_type: 1 00010000:00010000:1.0:1788572043.877015:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 238 lru_type: 1 00000080:00200000:1.0:1788572043.877023:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572043.877028:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572043.877032:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:1.0:1788572043.877052:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a41f8fb58 00010000:00010000:1.0:1788572043.877057:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 238 lru_type: 1 00010000:00010000:1.0:1788572043.877066:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 239 lru_type: 1 00000080:00200000:1.0:1788572043.877079:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572043.877086:0:62539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=24 00000080:00200000:1.0:1788572043.877091:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788572043.877104:0:62539:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=stripe_dir-9, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), flags=8 00000080:00200000:1.0:1788572043.877108:0:62539:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=stripe_dir-9, dir=[0x200009871:0x1:0x0](ffff94d4034900c8), intent=getattr 00000002:00010000:1.0:1788572043.877138:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: stripe_dir-9,[0x0:0x0:0x0]) in obj [0x200009871:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788572043.877176:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08f3 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.877185:0:62539:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08f3 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.877195:0:62539:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08f3 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572043.877270:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488430208:192.168.202.125@tcp:101:rm.0 00000100:00100000:1.0:1788572043.877341:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572043.881764:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488430208:192.168.202.125@tcp:101:rm.0 00010000:00010000:1.0:1788572043.881809:0:62539:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08f3 lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deee8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.881833:0:62539:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08f3 lrc: 4/1,0 mode: --/PR res: [0x200009871:0x15:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deee8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.881842:0:62539:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08f3 lrc: 4/1,0 mode: --/PR res: [0x200009871:0x15:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deee8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.881853:0:62539:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08f3 lrc: 4/1,0 mode: PR/PR res: [0x200009871:0x15:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deee8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.881863:0:62539:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08f3 lrc: 4/1,0 mode: PR/PR res: [0x200009871:0x15:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deee8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.881876:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08f3 lrc: 5/2,0 mode: PR/PR res: [0x200009871:0x15:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deee8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.881884:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08f3 lrc: 5/2,0 mode: PR/PR res: [0x200009871:0x15:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deee8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.881892:0:62539:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08f3 lrc: 4/1,0 mode: PR/PR res: [0x200009871:0x15:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deee8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572043.881910:0:62539:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d4049bd180 x1875450488430208/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 584/672 e 0 to 0 dl 1788572059 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572043.881933:0:62539:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08f3 lrc: 3/1,0 mode: PR/PR res: [0x200009871:0x15:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deee8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.881944:0:62539:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589973617/21 (0 0) 00000080:00200000:1.0:1788572043.881951:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200009871:0x1:0x0] suppgids 0 -1: rc 0 00000002:00010000:1.0:1788572043.882013:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009870:0xf:0x0]) in obj [0x200009870:0xf:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.882050:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 3/1,0 mode: --/CR res: [0x200009870:0xf:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.882054:0:62539:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 3/1,0 mode: --/CR res: [0x200009870:0xf:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.882061:0:62539:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 3/1,0 mode: --/CR res: [0x200009870:0xf:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572043.882075:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488430336:192.168.202.125@tcp:101:rm.0 00000100:00100000:1.0:1788572043.882140:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572043.888149:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488430336:192.168.202.125@tcp:101:rm.0 00010000:00010000:1.0:1788572043.888176:0:62539:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 4/1,0 mode: --/CR res: [0x200009870:0xf:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.888188:0:62539:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 4/1,0 mode: --/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.888217:0:62539:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 4/1,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.888266:0:62539:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 4/1,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.888278:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 5/2,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.888283:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 5/2,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.888288:0:62539:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 4/1,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572043.888296:0:62539:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d4049bd500 x1875450488430336/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/640 e 0 to 0 dl 1788572059 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572043.888306:0:62539:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 3/1,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.888314:0:62539:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589973616/15 (0 0) 00010000:00010000:1.0:1788572043.888321:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 3/1,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.888326:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 2/0,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:1.0:1788572043.888349:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2400090a2:0xd:0x0]) in obj [0x2400090a2:0xd:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.888382:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 3/1,0 mode: --/CR res: [0x2400090a2:0xd:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.888387:0:62539:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 3/1,0 mode: --/CR res: [0x2400090a2:0xd:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.888393:0:62539:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 3/1,0 mode: --/CR res: [0x2400090a2:0xd:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572043.888408:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488430464:192.168.202.125@tcp:101:rm.0 00000100:00100000:1.0:1788572043.888479:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572043.896592:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488430464:192.168.202.125@tcp:101:rm.0 00010000:00010000:1.0:1788572043.896616:0:62539:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 4/1,0 mode: --/CR res: [0x2400090a2:0xd:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4def04 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.896626:0:62539:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 4/1,0 mode: --/PR res: [0x2400090a2:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4def04 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.896634:0:62539:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 4/1,0 mode: PR/PR res: [0x2400090a2:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4def04 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.896641:0:62539:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 4/1,0 mode: PR/PR res: [0x2400090a2:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4def04 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.896650:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 5/2,0 mode: PR/PR res: [0x2400090a2:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4def04 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.896655:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 5/2,0 mode: PR/PR res: [0x2400090a2:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4def04 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.896664:0:62539:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 4/1,0 mode: PR/PR res: [0x2400090a2:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4def04 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572043.896671:0:62539:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d4049bd500 x1875450488430464/t0(0) o101->lustre-MDT0001-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/640 e 0 to 0 dl 1788572059 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572043.896680:0:62539:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4def04 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.896688:0:62539:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d404a46c00 type 13 mode 4 res 9663713442/13 (0 0) 00010000:00010000:1.0:1788572043.896693:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4def04 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.896699:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4def04 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572043.898818:0:62539:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200009871:0x15:0x0], updating i_size 8192 00000080:00200000:1.0:1788572043.898825:0:62539:0:(namei.c:147:ll_iget()) got inode: [0x200009871:0x15:0x0](ffff94d410bc3248): rc = 0 00000080:00010000:1.0:1788572043.898838:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200009871:0x15:0x0](ffff94d410bc3248) for lock 0x1e260726fbbc08f3 00000080:00010000:1.0:1788572043.898859:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200009871:0x15:0x0](ffff94d410bc3248) 00000080:00010000:1.0:1788572043.898862:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc08f3 from it ffffb68a41f8fb20 00010000:00010000:1.0:1788572043.898867:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08f3 lrc: 3/1,0 mode: PR/PR res: [0x200009871:0x15:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deee8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.898876:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08f3 lrc: 2/0,0 mode: PR/PR res: [0x200009871:0x15:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deee8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:1.0:1788572043.898927:0:62539:0:(file.c:6285:ll_getattr_dentry()) START file stripe_dir-9:[0x200009871:0x15:0x0](ffff94d410bc3248), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572043.898932:0:62539:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200009871:0x15:0x0](ffff94d410bc3248),name=stripe_dir-9 00000002:00010000:1.0:1788572043.898951:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009871:0x15:0x0]) in obj [0x200009871:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.898965:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08f3 lrc: 2/1,0 mode: PR/PR res: [0x200009871:0x15:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deee8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572043.898976:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08f3 lrc: 2/1,0 mode: PR/PR res: [0x200009871:0x15:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deee8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:1.0:1788572043.898986:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009870:0xf:0x0]) in obj [0x200009870:0xf:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.898993:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 2/1,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572043.898999:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 2/1,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572043.899004:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 3/1,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572043.899008:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 2/0,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:1.0:1788572043.899015:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2400090a2:0xd:0x0]) in obj [0x2400090a2:0xd:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.899019:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0xd:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4def04 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572043.899024:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4def04 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572043.899029:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4def04 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572043.899033:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4def04 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572043.899039:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200009871:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1788572043.899043:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200009871:0x15:0x0](ffff94d410bc3248) 00000080:00010000:1.0:1788572043.899045:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200009871:0x15:0x0](ffff94d410bc3248) for lock 0x1e260726fbbc08f3 00000080:00010000:1.0:1788572043.899047:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc08f3 from it ffffb68a41f8fcb8 00010000:00010000:1.0:1788572043.899048:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08f3 lrc: 3/1,0 mode: PR/PR res: [0x200009871:0x15:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deee8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572043.899053:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08f3 lrc: 2/0,0 mode: PR/PR res: [0x200009871:0x15:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deee8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:1.0:1788572043.899059:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009870:0xf:0x0]) in obj [0x200009870:0xf:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.899062:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 2/1,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572043.899066:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 2/1,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572043.899070:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 3/1,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572043.899074:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 2/0,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:1.0:1788572043.899078:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2400090a2:0xd:0x0]) in obj [0x2400090a2:0xd:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.899081:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0xd:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4def04 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572043.899085:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4def04 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572043.899089:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4def04 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572043.899092:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4def04 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788572043.899134:0:62539:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572043.899139:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.899143:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 239 lru_type: 1 00010000:00010000:1.0:1788572043.899147:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 239 lru_type: 1 00000080:00200000:1.0:1788572043.899150:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572043.899152:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572043.899154:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:1.0:1788572043.899156:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a41f8fbf8 00010000:00010000:1.0:1788572043.899158:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 239 lru_type: 1 00010000:00010000:1.0:1788572043.899166:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 240 lru_type: 1 00000080:00200000:1.0:1788572043.899173:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572043.899566:0:62539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=26 00000080:00200000:1.0:1788572043.899573:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788572043.899579:0:62539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=stripe_dir-9, flags=266 00000080:00200000:1.0:1788572043.899583:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x15:0x0](ffff94d410bc3248), inode mode 41ed mask 44 00000080:00200008:1.0:1788572043.899591:0:62539:0:(file.c:1090:ll_file_open()) START file stripe_dir-9:[0x200009871:0x15:0x0](ffff94d410bc3248), flags 704400 00000002:00010000:1.0:1788572043.899607:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009871:0x15:0x0]) in obj [0x200009871:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:1.0:1788572043.899660:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0908 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.899669:0:62539:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0908 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572043.899682:0:62539:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572043.899693:0:62539:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d4049bd180 x1875450488430720/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572043.899703:0:62539:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0908 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572043.899726:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488430720:192.168.202.125@tcp:101:rm.0 00000100:00100000:1.0:1788572043.899921:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572043.910630:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488430720:192.168.202.125@tcp:101:rm.0 00010000:00010000:1.0:1788572043.910653:0:62539:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0908 lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.910673:0:62539:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0908 lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.910679:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0908 lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.910685:0:62539:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0908 lrc: 3/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.910693:0:62539:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0908 lrc: 4/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.910699:0:62539:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412583e00 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:1.0:1788572043.910703:0:62539:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0908 lrc: 4/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.910709:0:62539:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0908 lrc: 4/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.910716:0:62539:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0908 lrc: 5/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.910723:0:62539:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572043.910729:0:62539:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0908 lrc: 3/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572043.910735:0:62539:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0908 lrc: 0/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 62539 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572043.910752:0:62539:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff94d4049bd180 x1875450488430720/t85899346162(85899346162) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/696 e 0 to 0 dl 1788572059 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572043.910770:0:62539:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d4049bd180 x1875450488430720/t85899346162(85899346162) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/696 e 0 to 0 dl 1788572059 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572043.910785:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200009871:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572043.910828:0:62539:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200009871:0x15:0x0], updating i_size 4096 00000002:00100000:1.0:1788572043.910847:0:62539:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d4049bd180 x1875450488430720/t85899346162(85899346162) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 608/696 e 0 to 0 dl 1788572059 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1788572043.911026:0:62539:0:(file.c:6285:ll_getattr_dentry()) START file stripe_dir-9:[0x200009871:0x15:0x0](ffff94d410bc3248), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572043.911032:0:62539:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200009871:0x15:0x0](ffff94d410bc3248),name=stripe_dir-9 00000002:00010000:1.0:1788572043.911048:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009871:0x15:0x0]) in obj [0x200009871:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.911060:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08f3 lrc: 2/1,0 mode: PR/PR res: [0x200009871:0x15:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deee8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572043.911069:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08f3 lrc: 2/1,0 mode: PR/PR res: [0x200009871:0x15:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deee8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:1.0:1788572043.911075:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009870:0xf:0x0]) in obj [0x200009870:0xf:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.911079:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 2/1,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572043.911083:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 2/1,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572043.911088:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 3/1,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572043.911093:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 2/0,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00010000:1.0:1788572043.911104:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2400090a2:0xd:0x0]) in obj [0x2400090a2:0xd:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.911109:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0xd:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4def04 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572043.911114:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4def04 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572043.911118:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4def04 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572043.911122:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4def04 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788572043.911127:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200009871:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1788572043.911132:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200009871:0x15:0x0](ffff94d410bc3248) 00000080:00010000:1.0:1788572043.911134:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200009871:0x15:0x0](ffff94d410bc3248) for lock 0x1e260726fbbc08f3 00000080:00010000:1.0:1788572043.911136:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc08f3 from it ffffb68a41f8fcd8 00010000:00010000:1.0:1788572043.911138:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08f3 lrc: 3/1,0 mode: PR/PR res: [0x200009871:0x15:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deee8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572043.911143:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08f3 lrc: 2/0,0 mode: PR/PR res: [0x200009871:0x15:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deee8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:1.0:1788572043.911149:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009870:0xf:0x0]) in obj [0x200009870:0xf:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.911151:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 2/1,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572043.911159:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 2/1,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572043.911163:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 3/1,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572043.911167:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 2/0,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000002:00010000:1.0:1788572043.911170:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2400090a2:0xd:0x0]) in obj [0x2400090a2:0xd:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.911173:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0xd:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4def04 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572043.911176:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4def04 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572043.911180:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4def04 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572043.911184:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4def04 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1788572043.912546:0:62539:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200009871:0x15:0x0](ffff94d410bc3248) pos/size0/8192 32bit_api 0 00010000:00010000:1.0:1788572043.912569:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582a00/0x1e260726fbbc0087 lrc: 2/0,0 mode: PR/PR res: [0x200009871:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd795 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00000002:00010000:1.0:1788572043.912601:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009870:0xf:0x0]) in obj [0x200009870:0xf:0x0], intent: readdir flags 0 00010000:00010000:1.0:1788572043.912608:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 2/1,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572043.912613:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 2/1,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000100:00100000:1.0:1788572043.913141:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488430848:192.168.202.125@tcp:37:rm.0 00000100:00100000:1.0:1788572043.913298:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572043.917897:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488430848:192.168.202.125@tcp:37:rm.0 00010000:00010000:1.0:1788572043.918691:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 3/1,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572043.918703:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 2/0,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00010000:1.0:1788572043.918721:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2400090a2:0xd:0x0]) in obj [0x2400090a2:0xd:0x0], intent: readdir flags 0 00010000:00010000:1.0:1788572043.918733:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0xd:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4def04 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572043.918742:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 2/1,0 mode: PR/PR res: [0x2400090a2:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4def04 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000100:00100000:1.0:1788572043.919455:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488430976:192.168.202.125@tcp:37:rm.0 00000100:00100000:1.0:1788572043.919580:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572043.924160:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488430976:192.168.202.125@tcp:37:rm.0 00010000:00010000:1.0:1788572043.924717:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 3/1,0 mode: PR/PR res: [0x2400090a2:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4def04 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572043.924728:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0901 lrc: 2/0,0 mode: PR/PR res: [0x2400090a2:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4def04 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1788572043.924783:0:62539:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200009871:0x15:0x0](ffff94d410bc3248) pos/size18446744073709551614/8192 32bit_api 0 00000080:00200008:1.0:1788572043.924798:0:62539:0:(file.c:479:ll_file_release()) START file stripe_dir-9:[0x200009871:0x15:0x0](ffff94d410bc3248), flags 704000 00000002:00100000:1.0:1788572043.924847:0:62539:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff94d4049bd180 x1875450488430720/t85899346162(85899346162) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 608/696 e 0 to 0 dl 1788572059 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572043.924855:0:62539:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572043.924859:0:62539:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d4049bd500 x1875450488431232/t0(0) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572043.924875:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488431232:192.168.202.125@tcp:35:rm.0 00000100:00100000:1.0:1788572043.925001:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572043.926713:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488431232:192.168.202.125@tcp:35:rm.0 00000002:00100000:1.0:1788572043.926737:0:62539:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff94d4049bd180 x1875450488430720/t85899346162(85899346162) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 608/696 e 0 to 0 dl 1788572059 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572043.926798:0:62539:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572043.926811:0:62539:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.926825:0:62539:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 240 lru_type: 1 00010000:00010000:1.0:1788572043.926834:0:62539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 240 lru_type: 1 00000080:00200000:1.0:1788572043.926839:0:62539:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572043.926844:0:62539:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572043.926846:0:62539:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:1.0:1788572043.926851:0:62539:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a41f8fc10 00010000:00010000:1.0:1788572043.926854:0:62539:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 240 lru_type: 1 00010000:00010000:1.0:1788572043.926859:0:62539:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 241 lru_type: 1 00000080:00200000:1.0:1788572043.926868:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572043.926873:0:62539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=24 00000080:00200000:1.0:1788572043.926877:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788572043.926881:0:62539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=stripe_dir-9, flags=0 00000080:00200000:1.0:1788572043.926885:0:62539:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8), inode mode 41ed mask 3 00000080:00200000:1.0:1788572043.926889:0:62539:0:(namei.c:2328:ll_rmdir()) VFS Op:name=stripe_dir-9, dir=[0x200009871:0x1:0x0](ffff94d4034900c8) 00010000:00010000:1.0:1788572043.926905:0:62539:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08f3 lrc: 2/0,0 mode: PR/PR res: [0x200009871:0x15:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4deee8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572043.927073:0:62539:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08f3 lrc: 1/0,0 mode: --/PR res: [0x200009871:0x15:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4deee8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572043.927079:0:62539:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572043.927081:0:62539:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc08f3 lrc: 0/0,0 mode: --/PR res: [0x200009871:0x15:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4deee8 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:1.0:1788572043.927098:0:62539:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788572043.927103:0:62539:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d4049bd500 x1875450488431488/t0(0) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572043.927118:0:62539:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488431488:192.168.202.125@tcp:36:rm.0 00000100:00100000:1.0:1788572043.927199:0:62539:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101a80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572043.928899:0:2397:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.125@tcp (source: 12345-192.168.202.125@tcp) 00010000:00010000:1.0:1788572043.929273:0:42449:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788572043.929279:0:42449:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 2/0,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572043.929285:0:42449:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412583400 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:1.0:1788572043.929288:0:42449:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 2/0,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572043.929294:0:42449:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 2/0,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572043.929300:0:42449:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 3/0,0 mode: PR/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572043.929386:0:42449:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 2/0,0 mode: --/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572043.929391:0:42449:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572043.929414:0:42449:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 1/0,0 mode: --/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572043.929418:0:42449:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc08fa lrc: 0/0,0 mode: --/PR res: [0x200009870:0xf:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xec901abfeb4deef6 expref: -99 pid: 62539 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572043.929429:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572043.929430:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572043.929431:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572043.940138:0:2397:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.125@tcp (source: 12345-192.168.202.125@tcp) 00000100:00100000:1.0:1788572043.951402:0:62539:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:62539:1875450488431488:192.168.202.125@tcp:36:rm.0 00000080:00200000:1.0:1788572043.951446:0:62539:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200009871:0x15:0x0](ffff94d410bc3248) 00000080:00200000:2.0:1788572043.957669:0:62552:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572043.957692:0:62552:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572043.957708:0:62552:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 241 lru_type: 1 00010000:00010000:2.0:1788572043.957721:0:62552:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 241 lru_type: 1 00000080:00200000:2.0:1788572043.957727:0:62552:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572043.957733:0:62552:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572043.957736:0:62552:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:2.0:1788572043.957741:0:62552:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a4209fb58 00010000:00010000:2.0:1788572043.957744:0:62552:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 241 lru_type: 1 00010000:00010000:2.0:1788572043.957749:0:62552:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 242 lru_type: 1 00000080:00200000:2.0:1788572043.957762:0:62552:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572043.957768:0:62552:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=8 00000080:00200008:2.0:1788572043.957776:0:62552:0:(file.c:6285:ll_getattr_dentry()) START file d120.replay-single:[0x200009871:0x1:0x0](ffff94d4034900c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572043.957781:0:62552:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8),name=d120.replay-single 00000002:00010000:2.0:1788572043.957791:0:62552:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200009871:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572043.957799:0:62552:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 30 res 8589973617/1 (0 0) 00010000:00010000:2.0:1788572043.957840:0:62552:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537200/0x1e260726fbbc090f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 62552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572043.957846:0:62552:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537200/0x1e260726fbbc090f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 62552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572043.957854:0:62552:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537200/0x1e260726fbbc090f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 62552 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572043.957876:0:62552:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job find:9bf287c0-f582-4b83-84c1-15e766218620:62552:1875450488431872:192.168.202.125@tcp:101:find.0 00000100:00100000:2.0:1788572043.958029:0:62552:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416ec56c0 going to sleep for 11 seconds 00000080:00200000:1.0:1788572043.971463:0:62554:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572043.971488:0:62554:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.971505:0:62554:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 244 lru_type: 1 00010000:00010000:1.0:1788572043.971520:0:62554:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 244 lru_type: 1 00000080:00200000:1.0:1788572043.971528:0:62554:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572043.971535:0:62554:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572043.971539:0:62554:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:1.0:1788572043.971546:0:62554:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a41fd7c20 00010000:00010000:1.0:1788572043.971550:0:62554:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 244 lru_type: 1 00010000:00010000:1.0:1788572043.971558:0:62554:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 245 lru_type: 1 00000080:00200000:1.0:1788572043.971571:0:62554:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572043.971578:0:62554:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=0 00000080:00200000:1.0:1788572043.971583:0:62554:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572043.971587:0:62554:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572043.971592:0:62554:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 245 lru_type: 1 00010000:00010000:1.0:1788572043.971597:0:62554:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 245 lru_type: 1 00000080:00200000:1.0:1788572043.971605:0:62554:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572043.971608:0:62554:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572043.971609:0:62554:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0080 00000080:00010000:1.0:1788572043.971611:0:62554:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0080 from it ffffb68a41fd7d90 00010000:00010000:1.0:1788572043.971614:0:62554:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 245 lru_type: 1 00010000:00010000:1.0:1788572043.971618:0:62554:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 246 lru_type: 1 00000080:00200000:1.0:1788572043.971622:0:62554:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 3 00000080:00200000:1.0:1788572043.971626:0:62554:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d120.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988) 00010000:00010000:1.0:1788572043.971641:0:62554:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddc000/0x1e260726fbbc0080 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dd779 expref: -99 pid: 62497 timeout: 0 lvb_type: 0 lru_score: 246 lru_type: 1 00010000:00010000:1.0:1788572043.971661:0:62554:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583000/0x1e260726fbbc00bf lrc: 2/0,0 mode: CR/CR res: [0x200009871:0x1:0x0].0x0 bits 0x5/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dd875 expref: -99 pid: 62500 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000002:00100000:1.0:1788572043.971698:0:62554:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff94d4125b1180 x1875450488361088/t85899346007(85899346007) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572056 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572043.971708:0:62554:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572043.971711:0:62554:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d4049bdf80 x1875450488432384/t0(0) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572043.971731:0:62554:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdf80 pname:cluuid:pid:xid:nid:opc:job rmdir:9bf287c0-f582-4b83-84c1-15e766218620:62554:1875450488432384:192.168.202.125@tcp:35:rmdir.0 00000100:00100000:1.0:1788572043.971864:0:62554:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101a80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572043.977277:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488432512:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572043.989302:0:62554:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job rmdir:9bf287c0-f582-4b83-84c1-15e766218620:62554:1875450488433024:192.168.202.125@tcp:36:rmdir.0 00010000:00010000:2.0:1788572044.933580:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572046.981847:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572049.032763:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572049.095862:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488433152:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572049.095957:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488433408:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572049.095991:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788572049.095998:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488433280:192.168.202.125@tcp:400:kworker.0 00000100:00080000:1.0:1788572049.102529:0:2403:0:(recover.c:205:ptlrpc_request_handle_notconn()) import lustre-MDT0000-mdc-ffff94d405cc3800 of lustre-MDT0000_UUID@192.168.202.125@tcp abruptly disconnected: reconnecting 00000100:00080000:1.0:1788572049.102537:0:2403:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:1.0:1788572049.102543:0:2403:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff94d405cc3800: Connection to lustre-MDT0000 (at 192.168.202.125@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:1.0:1788572049.102549:0:2403:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572049.102554:0:2403:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3171 00000100:00080000:1.0:1788572049.102573:0:2403:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572049.102585:0:2403:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572049.102613:0:2403:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488433536/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572049.102649:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488433280:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572049.102670:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488433408:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572049.103108:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488433536:192.168.202.125@tcp:38:ptlrpcd_00_00.0 00000100:00080000:1.0:1788572049.111158:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572049.111173:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-19) 00000100:00100000:1.0:1788572049.111178:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488433536:192.168.202.125@tcp:38:ptlrpcd_00_00.0 00010000:00010000:2.0:1788572057.221513:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572059.270296:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572059.333799:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572059.333808:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572059.333812:0:470:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572059.333822:0:470:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3207 00000100:00080000:1.0:1788572059.333831:0:470:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: tried all connections, increasing latency to 10s 00000100:00080000:1.0:1788572059.333838:0:470:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572059.333847:0:470:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572059.333868:0:470:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488434176/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572059.333932:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488434304:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572059.334015:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788572059.334023:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488434048:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572059.338602:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488434304:192.168.202.125@tcp:400:kworker.0 00000100:00080000:1.0:1788572059.350827:0:2403:0:(recover.c:205:ptlrpc_request_handle_notconn()) import MGC192.168.202.125@tcp of MGS@192.168.202.125@tcp abruptly disconnected: reconnecting 00000100:00080000:1.0:1788572059.350835:0:2403:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from FULL to DISCONN 00000100:02020000:1.0:1788572059.350840:0:2403:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.202.125@tcp: Connection to MGS (at 192.168.202.125@tcp) was lost; in progress operations using this service will fail 00000100:00080000:1.0:1788572059.371939:0:2403:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.202.125@tcp for MGC192.168.202.125@tcp not replayable, auto-deactivating 00000100:00080000:1.0:1788572059.371945:0:2403:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:1.0:1788572059.371955:0:2403:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff94d4049bc380 x1875450488433152/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572065 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572059.371969:0:2403:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff94d408c54380 x1875450488433664/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572070 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572059.371990:0:2403:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff94d4049bce00 x1875450488434048/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572075 ref 1 fl Rpc:RNQU/200/0 rc 0/-107 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:1.0:1788572059.372003:0:2403:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:1.0:1788572059.372006:0:2403:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:1.0:1788572059.372008:0:2403:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:1.0:1788572059.372012:0:2403:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572059.372017:0:2403:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3099 00000100:00080000:1.0:1788572059.372032:0:2403:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d40490a800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788572059.382191:0:2403:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488434432/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572059.382249:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488434048:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572059.382271:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488433152:192.168.202.125@tcp:400:kworker.0 10000000:01000000:1.0:1788572060.805388:0:42451:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788572060.805397:0:42451:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-cliir-ffff94d405cc3800 from 93 10000000:01000000:1.0:1788572060.805402:0:42451:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788572060.805455:0:42451:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412583000/0x1e260726fbbc0916 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572060.805468:0:42451:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.125@tcp lock: ffff94d412583000/0x1e260726fbbc0916 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572060.805479:0:42451:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.125@tcp lock: ffff94d412583000/0x1e260726fbbc0916 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572060.805508:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488434560:192.168.202.125@tcp:101:ll_cfg_requeue.0 00000100:00100000:1.0:1788572060.805670:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408d009c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572060.811390:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488434560:192.168.202.125@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788572060.811419:0:42451:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.125@tcp lock: ffff94d412583000/0x1e260726fbbc0916 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4df268 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572060.811433:0:42451:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.125@tcp lock: ffff94d412583000/0x1e260726fbbc0916 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xec901abfeb4df268 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572060.811508:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488434688:192.168.202.125@tcp:256:ll_cfg_requeue.0 00000100:00100000:1.0:1788572060.811651:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408d009c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572060.828670:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488434688:192.168.202.125@tcp:256:ll_cfg_requeue.0 00000100:00080000:1.0:1788572060.828733:0:42451:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff94d405bab000 already not connected (conn 48, was 0): DISCONN 00010000:00080000:1.0:1788572060.828746:0:42451:0:(ldlm_lib.c:85:import_set_conn()) imp ffff94d405bab000@lustre-MDT0000-mdc-ffff94d405cc3800: found existing conn 192.168.202.125@tcp, moved to head 00000100:00080000:1.0:1788572060.828752:0:42451:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572060.828757:0:42451:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 0 00000100:00080000:1.0:1788572060.828766:0:42451:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572060.828777:0:42451:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572060.828805:0:42451:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488434816/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 10000000:01000000:1.0:1788572060.828871:0:42451:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.202.125@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:1.0:1788572060.828892:0:42451:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412583000/0x1e260726fbbc0916 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4df268 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572060.828903:0:42451:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.125@tcp lock: ffff94d412583000/0x1e260726fbbc0916 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4df268 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572060.828912:0:42451:0:(mgc_request.c:611:do_requeue()) updating log lustre-client 10000000:01000000:1.0:1788572060.828916:0:42451:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-client-ffff94d405cc3800 from 52 10000000:01000000:1.0:1788572060.828919:0:42451:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:1.0:1788572060.828947:0:42451:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412583600/0x1e260726fbbc091d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572060.828955:0:42451:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.125@tcp lock: ffff94d412583600/0x1e260726fbbc091d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572060.828965:0:42451:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.125@tcp lock: ffff94d412583600/0x1e260726fbbc091d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572060.830723:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488434944:192.168.202.125@tcp:101:ll_cfg_requeue.0 00000100:00100000:1.0:1788572060.830791:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408d009c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572060.835826:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488434944:192.168.202.125@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788572060.835976:0:42451:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.125@tcp lock: ffff94d412583600/0x1e260726fbbc091d lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4df26f expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572060.835989:0:42451:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.125@tcp lock: ffff94d412583600/0x1e260726fbbc091d lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xec901abfeb4df26f expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572060.836069:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488435072:192.168.202.125@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788572060.836259:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408d009c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572060.843566:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488435072:192.168.202.125@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788572060.843635:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488435200:192.168.202.125@tcp:503:ll_cfg_requeue.0 00000100:00100000:1.0:1788572060.843715:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408d009c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572060.856037:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488435200:192.168.202.125@tcp:503:ll_cfg_requeue.0 00000040:00080000:1.0:1788572060.856731:0:62814:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 64768 count 51 00000020:01000000:2.0:1788572060.857231:0:42451:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 52-51 (rc=0) 10000000:01000000:2.0:1788572060.857261:0:42451:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.202.125@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:2.0:1788572060.857281:0:42451:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412583600/0x1e260726fbbc091d lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4df26f expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572060.857293:0:42451:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.125@tcp lock: ffff94d412583600/0x1e260726fbbc091d lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4df26f expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572060.857310:0:42451:0:(mgc_request.c:611:do_requeue()) updating log params 10000000:01000000:2.0:1788572060.857312:0:42451:0:(mgc_request.c:1801:mgc_process_log()) Process log params-ffff94d405cc3800 from 7 10000000:01000000:2.0:1788572060.857315:0:42451:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:2.0:1788572060.857354:0:42451:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412537800/0x1e260726fbbc0924 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572060.857362:0:42451:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.125@tcp lock: ffff94d412537800/0x1e260726fbbc0924 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572060.857371:0:42451:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.125@tcp lock: ffff94d412537800/0x1e260726fbbc0924 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572060.857395:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488435328:192.168.202.125@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788572060.857477:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416ec5240 going to sleep for 11 seconds 00000100:00100000:2.0:1788572060.861618:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488435328:192.168.202.125@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788572060.861650:0:42451:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.125@tcp lock: ffff94d412537800/0x1e260726fbbc0924 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4df27d expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572060.861663:0:42451:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.125@tcp lock: ffff94d412537800/0x1e260726fbbc0924 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xec901abfeb4df27d expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572060.861743:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488435456:192.168.202.125@tcp:501:ll_cfg_requeue.0 00000100:00100000:2.0:1788572060.861831:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416ec5240 going to sleep for 11 seconds 00000100:00100000:2.0:1788572060.864644:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488435456:192.168.202.125@tcp:501:ll_cfg_requeue.0 00000100:00100000:2.0:1788572060.864705:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488435584:192.168.202.125@tcp:503:ll_cfg_requeue.0 00000100:00100000:2.0:1788572060.864796:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416ec5240 going to sleep for 11 seconds 00010000:00080000:1.0:1788572060.879715:0:2402:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff94d4049bce00 x1875450488434816/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/416 e 0 to 0 dl 1788572070 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572060.879743:0:2402:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to target with instance 21 00000100:00080000:1.0:1788572060.879767:0:2402:0:(import.c:1338:ptlrpc_connect_interpret()) @@@ lustre-MDT0000-mdc-ffff94d405cc3800: evicting (reconnect/recover flags not set: 4) req@ffff94d4049bce00 x1875450488434816/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/416 e 0 to 0 dl 1788572070 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572060.879776:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to EVICTED 00000100:02020000:1.0:1788572060.879782:0:2402:0:(import.c:1663:ptlrpc_import_recovery_state_machine()) lustre-MDT0000-mdc-ffff94d405cc3800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. 00000040:00080000:2.0:1788572060.890925:0:62815:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 64768 count 7 00000100:00080000:1.0:1788572060.891107:0:2402:0:(import.c:1674:ptlrpc_import_recovery_state_machine()) evicted from lustre-MDT0000_UUID@192.168.202.125@tcp; invalidating 00000100:00100000:1.0:1788572060.892498:0:62816:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00010000:00010000:1.0:1788572060.892561:0:62816:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481e200/0x1e260726fbbbfe50 lrc: 3/0,0 mode: PR/PR res: [0x2000090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0xec901abfeb4dd0db expref: -99 pid: 61836 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.892599:0:62816:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572060.892611:0:62816:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481e200/0x1e260726fbbbfe50 lrc: 0/0,0 mode: --/PR res: [0x2000090a0:0x3:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0xec901abfeb4dd0db expref: -99 pid: 61836 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.892648:0:62816:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481e000/0x1e260726fbbbfe6c lrc: 3/0,0 mode: PR/PR res: [0x2000090a0:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0xec901abfeb4dd12f expref: -99 pid: 61837 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.892660:0:62816:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572060.892665:0:62816:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481e000/0x1e260726fbbbfe6c lrc: 0/0,0 mode: --/PR res: [0x2000090a0:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0xec901abfeb4dd12f expref: -99 pid: 61837 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.892694:0:62816:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddda00/0x1e260726fbbbfe88 lrc: 3/0,0 mode: PR/PR res: [0x2000090a0:0x5:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0xec901abfeb4dd183 expref: -99 pid: 61838 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.892706:0:62816:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572060.892711:0:62816:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d408ddda00/0x1e260726fbbbfe88 lrc: 0/0,0 mode: --/PR res: [0x2000090a0:0x5:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0xec901abfeb4dd183 expref: -99 pid: 61838 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.892740:0:62816:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a3400/0x1e260726fbbbfea4 lrc: 3/0,0 mode: PR/PR res: [0x2000090a0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0xec901abfeb4dd1d7 expref: -99 pid: 61839 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.892751:0:62816:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572060.892755:0:62816:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a3400/0x1e260726fbbbfea4 lrc: 0/0,0 mode: --/PR res: [0x2000090a0:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0xec901abfeb4dd1d7 expref: -99 pid: 61839 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.892795:0:62816:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481e600/0x1e260726fbbbfec0 lrc: 3/0,0 mode: PR/PR res: [0x2000090a0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0xec901abfeb4dd22b expref: -99 pid: 61840 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.892806:0:62816:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572060.892810:0:62816:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481e600/0x1e260726fbbbfec0 lrc: 0/0,0 mode: --/PR res: [0x2000090a0:0x7:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0xec901abfeb4dd22b expref: -99 pid: 61840 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.892836:0:62816:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ec00/0x1e260726fbbbfedc lrc: 3/0,0 mode: PR/PR res: [0x2000090a0:0x8:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0xec901abfeb4dd27f expref: -99 pid: 61841 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.892847:0:62816:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572060.892850:0:62816:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ec00/0x1e260726fbbbfedc lrc: 0/0,0 mode: --/PR res: [0x2000090a0:0x8:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0xec901abfeb4dd27f expref: -99 pid: 61841 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.892877:0:62816:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ee00/0x1e260726fbbbfef8 lrc: 3/0,0 mode: PR/PR res: [0x2000090a0:0x9:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0xec901abfeb4dd2d3 expref: -99 pid: 61842 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.892887:0:62816:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572060.892891:0:62816:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481ee00/0x1e260726fbbbfef8 lrc: 0/0,0 mode: --/PR res: [0x2000090a0:0x9:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0xec901abfeb4dd2d3 expref: -99 pid: 61842 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.892918:0:62816:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a3800/0x1e260726fbbbff14 lrc: 3/0,0 mode: PR/PR res: [0x2000090a0:0xa:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0xec901abfeb4dd327 expref: -99 pid: 61843 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.892929:0:62816:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572060.892933:0:62816:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a3800/0x1e260726fbbbff14 lrc: 0/0,0 mode: --/PR res: [0x2000090a0:0xa:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0xec901abfeb4dd327 expref: -99 pid: 61843 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.892966:0:62816:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e600/0x1e260726fbbbff30 lrc: 3/0,0 mode: PR/PR res: [0x2000090a0:0xb:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0xec901abfeb4dd37b expref: -99 pid: 61844 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.892976:0:62816:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572060.892980:0:62816:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e600/0x1e260726fbbbff30 lrc: 0/0,0 mode: --/PR res: [0x2000090a0:0xb:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0xec901abfeb4dd37b expref: -99 pid: 61844 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.893006:0:62816:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2400/0x1e260726fbbbff4c lrc: 3/0,0 mode: PR/PR res: [0x2000090a0:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0xec901abfeb4dd3cf expref: -99 pid: 61845 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.893017:0:62816:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572060.893021:0:62816:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2400/0x1e260726fbbbff4c lrc: 0/0,0 mode: --/PR res: [0x2000090a0:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0xec901abfeb4dd3cf expref: -99 pid: 61845 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.893048:0:62816:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a3200/0x1e260726fbbbff68 lrc: 3/0,0 mode: PR/PR res: [0x2000090a0:0xd:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0xec901abfeb4dd423 expref: -99 pid: 61846 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.893066:0:62816:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572060.893070:0:62816:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a3200/0x1e260726fbbbff68 lrc: 0/0,0 mode: --/PR res: [0x2000090a0:0xd:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0xec901abfeb4dd423 expref: -99 pid: 61846 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.893100:0:62816:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e000/0x1e260726fbbbff84 lrc: 3/0,0 mode: PR/PR res: [0x2000090a0:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0xec901abfeb4dd477 expref: -99 pid: 61847 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.893109:0:62816:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572060.893113:0:62816:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251e000/0x1e260726fbbbff84 lrc: 0/0,0 mode: --/PR res: [0x2000090a0:0xe:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0xec901abfeb4dd477 expref: -99 pid: 61847 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.893137:0:62816:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2e00/0x1e260726fbbbffa0 lrc: 3/0,0 mode: PR/PR res: [0x2000090a0:0xf:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0xec901abfeb4dd4cb expref: -99 pid: 61848 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.893146:0:62816:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572060.893149:0:62816:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2e00/0x1e260726fbbbffa0 lrc: 0/0,0 mode: --/PR res: [0x2000090a0:0xf:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0xec901abfeb4dd4cb expref: -99 pid: 61848 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.893173:0:62816:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2800/0x1e260726fbbbffbc lrc: 3/0,0 mode: PR/PR res: [0x2000090a0:0x10:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0xec901abfeb4dd51f expref: -99 pid: 61849 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.893182:0:62816:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572060.893185:0:62816:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089a2800/0x1e260726fbbbffbc lrc: 0/0,0 mode: --/PR res: [0x2000090a0:0x10:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0xec901abfeb4dd51f expref: -99 pid: 61849 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.893243:0:62816:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ea00/0x1e260726fbbbffd8 lrc: 3/0,0 mode: PR/PR res: [0x2000090a0:0x11:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0xec901abfeb4dd573 expref: -99 pid: 61850 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.893255:0:62816:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572060.893260:0:62816:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251ea00/0x1e260726fbbbffd8 lrc: 0/0,0 mode: --/PR res: [0x2000090a0:0x11:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0xec901abfeb4dd573 expref: -99 pid: 61850 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.893290:0:62816:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583c00/0x1e260726fbbbfff4 lrc: 3/0,0 mode: PR/PR res: [0x2000090a0:0x12:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0xec901abfeb4dd5c7 expref: -99 pid: 61851 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.893301:0:62816:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572060.893305:0:62816:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583c00/0x1e260726fbbbfff4 lrc: 0/0,0 mode: --/PR res: [0x2000090a0:0x12:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0xec901abfeb4dd5c7 expref: -99 pid: 61851 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.893330:0:62816:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481e800/0x1e260726fbbc0010 lrc: 3/0,0 mode: PR/PR res: [0x2000090a0:0x13:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0xec901abfeb4dd61b expref: -99 pid: 61852 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.893340:0:62816:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572060.893343:0:62816:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481e800/0x1e260726fbbc0010 lrc: 0/0,0 mode: --/PR res: [0x2000090a0:0x13:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0xec901abfeb4dd61b expref: -99 pid: 61852 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.893368:0:62816:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481fc00/0x1e260726fbbc002c lrc: 3/0,0 mode: PR/PR res: [0x2000090a0:0x14:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0xec901abfeb4dd66f expref: -99 pid: 61853 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.893378:0:62816:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572060.893382:0:62816:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d40481fc00/0x1e260726fbbc002c lrc: 0/0,0 mode: --/PR res: [0x2000090a0:0x14:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0xec901abfeb4dd66f expref: -99 pid: 61853 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.893412:0:62816:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582800/0x1e260726fbbc0048 lrc: 3/0,0 mode: PR/PR res: [0x2000090a0:0x15:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0xec901abfeb4dd6c3 expref: -99 pid: 61854 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.893425:0:62816:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572060.893430:0:62816:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582800/0x1e260726fbbc0048 lrc: 0/0,0 mode: --/PR res: [0x2000090a0:0x15:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0xec901abfeb4dd6c3 expref: -99 pid: 61854 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.893463:0:62816:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089cac00/0x1e260726fbbc0064 lrc: 3/0,0 mode: PR/PR res: [0x2000090a0:0x16:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0xec901abfeb4dd717 expref: -99 pid: 61855 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.893475:0:62816:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572060.893480:0:62816:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d4089cac00/0x1e260726fbbc0064 lrc: 0/0,0 mode: --/PR res: [0x2000090a0:0x16:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0xec901abfeb4dd717 expref: -99 pid: 61855 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.893600:0:62816:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f600/0x1e260726fbbbfe42 lrc: 3/0,0 mode: PR/PR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0xec901abfeb4dd0b1 expref: -99 pid: 61836 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572060.893623:0:62816:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572060.893628:0:62816:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d41251f600/0x1e260726fbbbfe42 lrc: 0/0,0 mode: --/PR res: [0x2000061c1:0x1d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0xec901abfeb4dd0b1 expref: -99 pid: 61836 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00080000:1.0:1788572060.895232:0:62816:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from EVICTED to RECOVER 00000100:00080000:1.0:1788572060.895250:0:62816:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from RECOVER to FULL 00010000:00010000:2.0:1788572061.318351:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572063.365891:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572064.453388:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488436096:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572064.453490:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:1.0:1788572064.453498:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488435968:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572064.453530:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488436224:192.168.202.125@tcp:400:kworker.0 00010000:00010000:2.0:1788572064.478354:0:42449:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572064.478378:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572064.478381:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572064.478383:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572066.441705:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572068.493186:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572068.602626:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488436736:192.168.202.125@tcp:8:lfs.0 00000100:00100000:1.0:1788572068.608106:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-OST0000-osc-ffff94d405cc3800: committing for last_committed 0 gen 13 00000100:00080000:1.0:1788572068.608126:0:2402:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff94d405cc3800: connect to target with instance 1 00000100:00080000:1.0:1788572068.608136:0:2402:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff94d405cc3800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:1.0:1788572068.608150:0:2402:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:1.0:1788572068.608154:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bad000 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00000080:00000004:1.0:1788572068.608164:0:2402:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:1.0:1788572068.608169:0:2402:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:1.0:1788572068.608193:0:2402:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000ea98d549): req@ffff94d408d3ce00 x1875450488436864/t0(0) o13->lustre-OST0000-osc-ffff94d405cc3800@192.168.202.125@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572068.608254:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488436736:192.168.202.125@tcp:8:lfs.0 00000100:00100000:1.0:1788572068.610409:0:62867:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:62867:1875450488436864:192.168.202.125@tcp:13:lfs.0 00020000:00000004:1.0:1788572068.610435:0:62867:0:(obd_class.h:1117:obd_statfs()) lustre-OST0000-osc-ffff94d405cc3800: update ffff94d407dbd190 cache blocks 901369/958279 objects 261733/262144 00000080:00200080:1.0:1788572068.610505:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.610529:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00020000:00000004:1.0:1788572068.610536:0:62867:0:(obd_class.h:1091:obd_statfs()) lustre-OST0001-osc-ffff94d405cc3800: age 3177, max_age 3221 00000100:00080000:1.0:1788572068.610547:0:62867:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3177 00000100:00080000:1.0:1788572068.610556:0:62867:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d405cc3800: import ffff94d405bab800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572068.610566:0:62867:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8437760, lost: 0. 00000100:00100000:1.0:1788572068.610589:0:62867:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488436992/t0(0) o8->lustre-OST0001-osc-ffff94d405cc3800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572068.610630:0:62867:0:(client.c:1832:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff94d4049bc700 x1875450488437120/t0(0) o13->lustre-OST0001-osc-ffff94d405cc3800@192.168.202.125@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572068.610639:0:62867:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408d00480 going to sleep for 0 seconds 00000100:00100000:1.0:1788572068.621766:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-OST0001-osc-ffff94d405cc3800: committing for last_committed 0 gen 13 00000100:00080000:1.0:1788572068.621783:0:2402:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff94d405cc3800: connect to target with instance 1 00000100:00080000:1.0:1788572068.621800:0:2402:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff94d405cc3800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:1.0:1788572068.621812:0:2402:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:1.0:1788572068.621816:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab800 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00000080:00000004:1.0:1788572068.621826:0:2402:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:1.0:1788572068.621830:0:2402:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:1.0:1788572068.621845:0:2402:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000002cbb03d5): req@ffff94d4049bc700 x1875450488437120/t0(0) o13->lustre-OST0001-osc-ffff94d405cc3800@192.168.202.125@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572068.621870:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488436992:192.168.202.125@tcp:8:lfs.0 00000100:00100000:1.0:1788572068.632690:0:62867:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:62867:1875450488437120:192.168.202.125@tcp:13:lfs.0 00020000:00000004:1.0:1788572068.632718:0:62867:0:(obd_class.h:1117:obd_statfs()) lustre-OST0001-osc-ffff94d405cc3800: update ffff94d407dbb7d0 cache blocks 901369/958279 objects 261733/262144 00000080:00200080:1.0:1788572068.632800:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.632817:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.632824:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.632828:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.632833:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.632837:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.632841:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.632845:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.632850:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.632853:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.632857:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.632860:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.632864:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.632867:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.632872:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.632876:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.632880:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.632883:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.632887:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.632891:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.632895:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.632898:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.632903:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.632907:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.632911:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.632915:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.632919:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.632923:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.632955:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.632960:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.632964:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.632968:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.632971:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.632975:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.632979:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.632982:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.632987:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.632990:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.632994:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.632997:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633002:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633006:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633009:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633013:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633016:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633020:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633023:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633026:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633030:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633033:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633037:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633040:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633044:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633047:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633051:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633055:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633059:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633062:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633065:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633069:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633072:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633075:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633078:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633081:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633085:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633089:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633092:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633096:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633099:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633102:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633106:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633110:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633113:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633116:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633120:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633123:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633127:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633130:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633133:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633137:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633141:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633144:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633148:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633151:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633155:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633159:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633163:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633166:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633170:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633174:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633177:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633193:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633198:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633472:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633481:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633486:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633490:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633494:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633499:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633502:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633592:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633600:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633606:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633610:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633615:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633618:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633622:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633625:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633629:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633632:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633635:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633638:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633642:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633645:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633650:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633654:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633658:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633661:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633665:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633669:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633672:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633675:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633680:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633683:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572068.633686:0:62867:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe73057010 00020000:00000080:1.0:1788572068.633689:0:62867:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200008:1.0:1788572068.633701:0:62867:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 100000 00000100:00100000:2.0:1788572069.585037:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488437248:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572069.585120:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488437632:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572069.588055:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488437760:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572069.588077:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488437376:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572069.588110:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488437504:192.168.202.125@tcp:400:kworker.0 00010000:00010000:1.0:1788572070.536846:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572070.663747:0:62919:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788572070.663761:0:62919:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572070.663769:0:62919:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:3.0:1788572070.663773:0:62919:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:3.0:1788572070.663781:0:62919:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40d17cb8 00010000:00010000:3.0:1788572070.663785:0:62919:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572070.663794:0:62919:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:3.0:1788572070.664368:0:62919:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788572070.664373:0:62919:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:3.0:1788572070.664383:0:62919:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572070.664391:0:62919:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788572070.664401:0:62919:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1788572070.664408:0:62919:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572070.664413:0:62919:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:3.0:1788572070.664415:0:62919:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:3.0:1788572070.664419:0:62919:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40d17cb8 00010000:00010000:3.0:1788572070.664422:0:62919:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788572070.664433:0:62919:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788572070.664525:0:62919:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:3.0:1788572070.664533:0:62919:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572070.664540:0:62919:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788572070.664547:0:62919:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788572070.664553:0:62919:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572070.664556:0:62919:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:3.0:1788572070.664558:0:62919:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:3.0:1788572070.664561:0:62919:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40d17bb0 00010000:00010000:3.0:1788572070.664564:0:62919:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788572070.664571:0:62919:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1788572070.664579:0:62919:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 44 00000080:00200008:3.0:1788572070.664585:0:62919:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 100000 00000080:00200080:3.0:1788572070.664666:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:3.0:1788572070.664679:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f33c00 uarg=0000000000000000 00800000:00000004:3.0:1788572070.664685:0:62919:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0000-mdc-ffff94d405cc3800: age 3222, max_age 3223 00000100:00100000:3.0:1788572070.664725:0:62919:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:62919:1875450488437888:192.168.202.125@tcp:41:lfs.0 00000100:00100000:3.0:1788572070.664799:0:62919:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4125d46c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572070.672635:0:62919:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:62919:1875450488437888:192.168.202.125@tcp:41:lfs.0 00800000:00000004:1.0:1788572070.672666:0:62919:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0000-mdc-ffff94d405cc3800: update ffff94d412909e10 cache blocks 317140/353529 objects 1023608/1024000 00000080:00200080:1.0:1788572070.672867:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.672887:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00800000:00000004:1.0:1788572070.672895:0:62919:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0001-mdc-ffff94d405cc3800: age 3222, max_age 3223 00000100:00100000:1.0:1788572070.672932:0:62919:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:62919:1875450488438016:192.168.202.125@tcp:41:lfs.0 00000100:00100000:1.0:1788572070.673006:0:62919:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408d00180 going to sleep for 11 seconds 00000100:00100000:1.0:1788572070.676000:0:62919:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:62919:1875450488438016:192.168.202.125@tcp:41:lfs.0 00800000:00000004:1.0:1788572070.676028:0:62919:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0001-mdc-ffff94d405cc3800: update ffff94d407db9e10 cache blocks 317215/353529 objects 1023685/1024000 00000080:00200080:1.0:1788572070.676098:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676112:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676121:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676125:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676129:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676132:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676136:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676140:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676144:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676147:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676151:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676154:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676159:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676162:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676167:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676170:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676174:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676177:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676180:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676183:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676187:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676191:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676195:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676198:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676240:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676245:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676249:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676253:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676257:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676260:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676264:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676267:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676271:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676274:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676278:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676281:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676285:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676288:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676292:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676295:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676299:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676302:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676306:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676310:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676314:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676318:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676321:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676325:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676328:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676331:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676335:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676338:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676341:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676344:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676347:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676349:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676353:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676356:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676359:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676362:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676365:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676369:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676374:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676377:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676381:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676385:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676389:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676392:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676396:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676400:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676404:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676408:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676412:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676416:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676420:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676432:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676439:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676443:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676448:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676451:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676455:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676459:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676462:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676466:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676470:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676473:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676477:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676481:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676485:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676489:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676493:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676497:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676501:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676505:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676509:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676512:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676516:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676519:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676523:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676526:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676529:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676532:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676536:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676539:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676542:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676545:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676548:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676551:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676554:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676556:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676560:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676562:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676565:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676568:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676572:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676575:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676577:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676580:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676583:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676585:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676588:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676591:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676594:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676596:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676599:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00800000:00000080:1.0:1788572070.676602:0:62919:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.676605:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.676611:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00020000:00000004:1.0:1788572070.676617:0:62919:0:(obd_class.h:1091:obd_statfs()) lustre-OST0000-osc-ffff94d405cc3800: age 3222, max_age 3223 00000100:00100000:1.0:1788572070.676652:0:62919:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:62919:1875450488438144:192.168.202.125@tcp:13:lfs.0 00000100:00100000:1.0:1788572070.676839:0:62919:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408d00180 going to sleep for 11 seconds 00000100:00100000:1.0:1788572070.681552:0:62919:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:62919:1875450488438144:192.168.202.125@tcp:13:lfs.0 00020000:00000004:1.0:1788572070.681582:0:62919:0:(obd_class.h:1117:obd_statfs()) lustre-OST0000-osc-ffff94d405cc3800: update ffff94d407dbd190 cache blocks 901369/958279 objects 261733/262144 00000080:00200080:1.0:1788572070.681931:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.681947:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00020000:00000004:1.0:1788572070.681954:0:62919:0:(obd_class.h:1091:obd_statfs()) lustre-OST0001-osc-ffff94d405cc3800: age 3222, max_age 3223 00000100:00100000:1.0:1788572070.681985:0:62919:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:62919:1875450488438272:192.168.202.125@tcp:13:lfs.0 00000100:00100000:1.0:1788572070.682053:0:62919:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408d00180 going to sleep for 11 seconds 00000100:00100000:1.0:1788572070.687782:0:62919:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:62919:1875450488438272:192.168.202.125@tcp:13:lfs.0 00020000:00000004:1.0:1788572070.687814:0:62919:0:(obd_class.h:1117:obd_statfs()) lustre-OST0001-osc-ffff94d405cc3800: update ffff94d407dbb7d0 cache blocks 901369/958279 objects 261733/262144 00000080:00200080:1.0:1788572070.687900:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.687918:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.687925:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.687929:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.687934:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.687937:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.687941:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.687945:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.687949:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.687953:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.687957:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.687963:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.687967:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.687971:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.687975:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.687979:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.687983:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.687986:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.687989:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.687991:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.687995:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.687998:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688002:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688005:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688009:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688012:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688015:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688017:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688020:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688023:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688026:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688029:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688032:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688035:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688038:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688040:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688044:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688047:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688050:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688053:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688061:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688065:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688070:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688073:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688076:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688079:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688083:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688086:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688090:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688093:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688096:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688099:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688103:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688105:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688109:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688112:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688115:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688118:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688122:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688125:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688129:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688132:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688136:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688139:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688143:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688146:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688149:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688153:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688156:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688160:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688163:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688167:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688170:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688173:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688176:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688179:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688183:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688185:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688189:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688191:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688194:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688197:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688242:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688247:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688251:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688254:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688257:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688260:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688264:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688267:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688271:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688274:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688277:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688280:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688283:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688286:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688289:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688292:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688295:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688298:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688301:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688303:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688306:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688309:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688312:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688314:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688323:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688327:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688331:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688334:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688337:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688340:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688344:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688347:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688350:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688352:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688355:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688358:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688362:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688365:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688369:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688372:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688376:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688380:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200080:1.0:1788572070.688383:0:62919:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd97f3eb0 00020000:00000080:1.0:1788572070.688386:0:62919:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416e27400 uarg=0000000000000000 00000080:00200008:1.0:1788572070.688400:0:62919:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 100000 00010000:00010000:1.0:1788572072.584059:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572074.183418:0:62547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0939 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572074.183436:0:62547:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0939 lrc: 3/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572074.183529:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788572074.183550:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488438400:192.168.202.125@tcp:63:ll_sa.0 00010000:00010000:3.0:1788572074.191963:0:2406:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0939 lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT flags: 0x0 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572074.191983:0:2406:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0939 lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572074.191991:0:2406:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0939 lrc: 4/1,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572074.191999:0:2406:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0939 lrc: 3/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572074.192008:0:2406:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0939 lrc: 4/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572074.192015:0:2406:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412536c00 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:3.0:1788572074.192020:0:2406:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0939 lrc: 4/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x3 rrc: 2 type: IBT flags: 0x206400000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572074.192027:0:2406:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0939 lrc: 4/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x3 rrc: 2 type: IBT flags: 0x206400000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572074.192033:0:2406:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0939 lrc: 5/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x3 rrc: 2 type: IBT flags: 0x20e400000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572074.192041:0:2406:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572074.192047:0:2406:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0939 lrc: 3/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x3 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572074.192054:0:2406:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0939 lrc: 0/0,0 mode: --/CR res: [0x200009871:0x1:0x0].0x0 bits 0x3/0x3 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 62547 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572074.192089:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488438400:192.168.202.125@tcp:63:ll_sa.0 00000080:00200000:2.0:1788572074.317645:0:62547:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200009871:0x1:0x0](ffff94d4034900c8) 00010000:00010000:1.0:1788572074.629972:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572074.693513:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488438656:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572074.693600:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488438912:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572074.693630:0:2405:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:2.0:1788572074.693635:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488438528:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572074.693662:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488438784:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572074.693695:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488439040:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572074.698695:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488438528:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572074.698721:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488438784:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572074.698744:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488439040:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572074.698755:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488438656:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572074.698766:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488438912:192.168.202.125@tcp:400:kworker.0 00010000:00010000:1.0:1788572076.678071:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572078.728636:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572079.824594:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488439168:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572079.824608:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488439296:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572079.829830:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488439424:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572079.829870:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488439680:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572079.834623:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488439552:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572079.847916:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488439296:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572079.847935:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488439552:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572079.848813:0:62287:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job bash:9bf287c0-f582-4b83-84c1-15e766218620:62287:1875450488439808:192.168.202.125@tcp:101:bash.0 00010000:00010000:1.0:1788572079.848833:0:62287:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0940 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 62287 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572079.848846:0:62287:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0940 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 62287 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572079.848854:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0940 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 62287 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572079.848862:0:62287:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0940 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 62287 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572079.848871:0:62287:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0940 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 62287 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572079.848878:0:62287:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412167600 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:1.0:1788572079.848883:0:62287:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0940 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 62287 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572079.848890:0:62287:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0940 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 62287 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572079.848897:0:62287:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0940 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 62287 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572079.848904:0:62287:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572079.848912:0:62287:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0940 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 62287 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572079.848918:0:62287:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0940 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 62287 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572079.848936:0:62287:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff94d408d3ea00 x1875450488439808/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 592/584 e 0 to 0 dl 1788572095 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'bash.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572079.848952:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1788572079.848984:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:1.0:1788572079.848998:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00010000:1.0:1788572079.849008:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da38d0 00010000:00010000:1.0:1788572079.849012:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:1.0:1788572079.849019:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:1.0:1788572079.849171:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.849318:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.849328:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:1.0:1788572079.849336:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:1.0:1788572079.849349:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.849355:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.849359:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.849364:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.849367:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:1.0:1788572079.850085:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:1.0:1788572079.850104:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.850114:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.850668:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.850689:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.850702:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:1.0:1788572079.850714:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:1.0:1788572079.850721:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.850727:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.850731:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.850736:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.850740:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:1.0:1788572079.850747:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:1.0:1788572079.850757:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.850761:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.850800:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.850807:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.850814:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:1.0:1788572079.850822:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:1.0:1788572079.850829:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.850833:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.850836:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.850840:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.850843:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:1.0:1788572079.850852:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:1.0:1788572079.850860:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.850864:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.850906:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.850913:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.850919:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1788572079.850927:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:1.0:1788572079.850932:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.850936:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.850938:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.850942:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.850945:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1788572079.850952:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:1.0:1788572079.852128:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.852138:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.852621:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.852642:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.852659:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:1.0:1788572079.852672:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:1.0:1788572079.852681:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.852687:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.852690:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.852696:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.852700:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:1.0:1788572079.852709:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:1.0:1788572079.852718:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.852723:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.852778:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.852786:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.852793:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:1.0:1788572079.852807:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:1.0:1788572079.852815:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.852820:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.852822:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.852827:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.852831:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:1.0:1788572079.852839:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:1.0:1788572079.852847:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.852851:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.852881:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.852889:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.852895:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:1.0:1788572079.852904:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:1.0:1788572079.852911:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.852915:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.852917:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.852921:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.852925:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:1.0:1788572079.852933:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:1.0:1788572079.852942:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.852946:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.852985:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.852993:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.853000:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:1.0:1788572079.853010:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:1.0:1788572079.853017:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.853021:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.853023:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.853027:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.853030:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:1.0:1788572079.853037:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:1.0:1788572079.853046:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.853050:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.856649:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.856681:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.856700:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:1.0:1788572079.856716:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:1.0:1788572079.856724:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.856733:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.856737:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.856744:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.856747:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:1.0:1788572079.856756:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00200000:1.0:1788572079.856768:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.856776:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.856836:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.856844:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.856851:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:1.0:1788572079.856859:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00200000:1.0:1788572079.856865:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.856869:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.856871:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.856874:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.856877:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:1.0:1788572079.856883:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00200000:1.0:1788572079.856890:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.856894:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.856922:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.856929:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.856934:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:1.0:1788572079.856942:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00200000:1.0:1788572079.856948:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.856952:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.856955:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.856958:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.856961:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:1.0:1788572079.856967:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200000:1.0:1788572079.856978:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.856982:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.857020:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.857027:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.857033:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:1.0:1788572079.857042:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200000:1.0:1788572079.857047:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.857051:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.857053:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.857056:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.857059:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:1.0:1788572079.857078:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00200000:1.0:1788572079.857086:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.857089:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.857118:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.857124:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.857131:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:1.0:1788572079.857143:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00200000:1.0:1788572079.857149:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.857154:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.857156:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.857159:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.857162:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:1.0:1788572079.857169:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000080:00200000:1.0:1788572079.857177:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.857194:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.857259:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.857267:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.857274:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:1.0:1788572079.857282:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000080:00200000:1.0:1788572079.857288:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.857292:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.857294:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.857298:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.857300:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:1.0:1788572079.857306:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000080:00200000:1.0:1788572079.857314:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.857317:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.857354:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.857362:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.857368:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:1.0:1788572079.858026:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000080:00200000:1.0:1788572079.858037:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.858044:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.858047:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.858052:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.858055:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:1.0:1788572079.858063:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000080:00200000:1.0:1788572079.858073:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.858078:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.858135:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.858144:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.858152:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:1.0:1788572079.858160:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000080:00200000:1.0:1788572079.858168:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.858172:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.858175:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000100:00100000:2.0:1788572079.858340:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488439680:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572079.858367:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488439168:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572079.858379:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488439424:192.168.202.125@tcp:400:kworker.0 00000080:00010000:1.0:1788572079.858472:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.858488:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:1.0:1788572079.858497:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000080:00200000:1.0:1788572079.858506:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.858511:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.858545:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.858554:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.858561:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:1.0:1788572079.858570:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000080:00200000:1.0:1788572079.858577:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.858581:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.858584:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.858588:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.858591:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:1.0:1788572079.858598:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000080:00200000:1.0:1788572079.858607:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.858611:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.858650:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.858657:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.858663:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:1.0:1788572079.858670:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000080:00200000:1.0:1788572079.858675:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.858679:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.858681:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.858684:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.858687:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:1.0:1788572079.858694:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000080:00200000:1.0:1788572079.858707:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.858711:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.858738:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.858744:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.858751:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:1.0:1788572079.858758:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000080:00200000:1.0:1788572079.858764:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.858768:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.858770:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.858773:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.858776:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:1.0:1788572079.858782:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000080:00200000:1.0:1788572079.858788:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.858792:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.858829:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.858836:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.858842:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:1.0:1788572079.858854:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000080:00200000:1.0:1788572079.858860:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.858863:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.858865:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.858869:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.858871:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:1.0:1788572079.858879:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000080:00200000:1.0:1788572079.858886:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.858889:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.858917:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.858925:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.858930:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:1.0:1788572079.858939:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000080:00200000:1.0:1788572079.858945:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.858948:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.858951:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.858955:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.858958:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:1.0:1788572079.858965:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000080:00200000:1.0:1788572079.858972:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.858975:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.859011:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.859018:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.859024:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:1.0:1788572079.859032:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000080:00200000:1.0:1788572079.859038:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.859042:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.859044:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.859047:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.859050:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:1.0:1788572079.859058:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:1.0:1788572079.859066:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.859068:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.859095:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.859101:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.859107:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:1.0:1788572079.859115:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:1.0:1788572079.859121:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.859125:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.859127:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.859130:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.859132:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:1.0:1788572079.859139:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000080:00200000:1.0:1788572079.859146:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.859149:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.863474:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.863516:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.863534:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00010000:00010000:1.0:1788572079.863550:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000080:00200000:1.0:1788572079.863559:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.863568:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.863572:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.863579:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.863583:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00010000:00010000:1.0:1788572079.863592:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00000080:00200000:1.0:1788572079.863607:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.863614:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.863713:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.863722:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.863730:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00010000:00010000:1.0:1788572079.863738:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00000080:00200000:1.0:1788572079.863745:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.863750:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.863753:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.863757:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.863760:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00010000:00010000:1.0:1788572079.863769:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00000080:00200000:1.0:1788572079.863781:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.863785:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.863840:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.863848:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.863855:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00010000:00010000:1.0:1788572079.863864:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00000080:00200000:1.0:1788572079.863871:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.863875:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.863878:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.863881:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.863885:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00010000:00010000:1.0:1788572079.863893:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00000080:00200000:1.0:1788572079.863901:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.863905:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.863935:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.863942:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.863949:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00010000:00010000:1.0:1788572079.863963:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00000080:00200000:1.0:1788572079.863970:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.863973:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.863976:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.863980:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.863983:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00010000:00010000:1.0:1788572079.863990:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00000080:00200000:1.0:1788572079.863997:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.864000:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.864037:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.864044:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.864050:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00010000:00010000:1.0:1788572079.864059:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00000080:00200000:1.0:1788572079.864065:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.864069:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.864071:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.864074:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.864078:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00010000:00010000:1.0:1788572079.864084:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00000080:00200000:1.0:1788572079.864092:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.864095:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.864122:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.864128:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.864134:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00010000:00010000:1.0:1788572079.864142:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00000080:00200000:1.0:1788572079.864149:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.864153:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.864155:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.864158:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.864161:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00010000:00010000:1.0:1788572079.864169:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00000080:00200000:1.0:1788572079.864177:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.864198:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.864264:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.864271:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.864278:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00010000:00010000:1.0:1788572079.864286:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00000080:00200000:1.0:1788572079.864292:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.864296:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.864298:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.864302:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.864305:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00010000:00010000:1.0:1788572079.864312:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00000080:00200000:1.0:1788572079.864320:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.864324:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.864351:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.864358:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.864364:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00010000:00010000:1.0:1788572079.864373:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00000080:00200000:1.0:1788572079.864379:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.864383:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.864385:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.864389:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.864392:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00010000:00010000:1.0:1788572079.864399:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00000080:00200000:1.0:1788572079.864406:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.864410:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.864446:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.864453:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.864460:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00010000:00010000:1.0:1788572079.864467:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00000080:00200000:1.0:1788572079.864473:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.864477:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.864490:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.864494:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.864497:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00010000:00010000:1.0:1788572079.864505:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00000080:00200000:1.0:1788572079.864517:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.864521:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.864549:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.864557:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.864564:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00010000:00010000:1.0:1788572079.864572:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00000080:00200000:1.0:1788572079.864578:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.864582:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.864584:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.864588:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.864591:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00010000:00010000:1.0:1788572079.864599:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00000080:00200000:1.0:1788572079.864606:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.864609:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.864646:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.864653:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.864659:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00010000:00010000:1.0:1788572079.864673:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00000080:00200000:1.0:1788572079.864680:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.864684:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.864687:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.864690:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.864693:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00010000:00010000:1.0:1788572079.864700:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00000080:00200000:1.0:1788572079.864707:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.864711:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.864737:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.864744:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.864750:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00010000:00010000:1.0:1788572079.864758:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00000080:00200000:1.0:1788572079.864764:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.864768:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.864770:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.864774:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.864778:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00010000:00010000:1.0:1788572079.864785:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00000080:00200000:1.0:1788572079.864793:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.864796:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.864833:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.864840:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.864846:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00010000:00010000:1.0:1788572079.864854:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00000080:00200000:1.0:1788572079.864860:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.864864:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.864867:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.864870:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.864873:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00010000:00010000:1.0:1788572079.864879:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00000080:00200000:1.0:1788572079.864887:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.864890:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.864918:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.864925:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.864931:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00010000:00010000:1.0:1788572079.864939:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00000080:00200000:1.0:1788572079.864945:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.864948:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.864950:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.864953:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.864956:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00010000:00010000:1.0:1788572079.864963:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 45 lru_type: 1 00000080:00200000:1.0:1788572079.864971:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.864974:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.865011:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.865017:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.865023:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 45 lru_type: 1 00010000:00010000:1.0:1788572079.865031:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 45 lru_type: 1 00000080:00200000:1.0:1788572079.865037:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.865042:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.865044:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.865047:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.865050:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 45 lru_type: 1 00010000:00010000:1.0:1788572079.865058:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 46 lru_type: 1 00000080:00200000:1.0:1788572079.865066:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.865069:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.865092:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.865098:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.865104:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 46 lru_type: 1 00010000:00010000:1.0:1788572079.865112:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 46 lru_type: 1 00000080:00200000:1.0:1788572079.865118:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.865122:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.865124:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.865127:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3b58 00010000:00010000:1.0:1788572079.865130:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 46 lru_type: 1 00010000:00010000:1.0:1788572079.865137:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 47 lru_type: 1 00000080:00200000:1.0:1788572079.865149:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.865152:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=25 00000080:00200000:1.0:1788572079.865313:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572079.865321:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572079.865327:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 47 lru_type: 1 00010000:00010000:1.0:1788572079.865335:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 47 lru_type: 1 00000080:00200000:1.0:1788572079.865340:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572079.865343:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572079.865346:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:1.0:1788572079.865349:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3bf8 00010000:00010000:1.0:1788572079.865351:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 47 lru_type: 1 00010000:00010000:1.0:1788572079.865358:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 48 lru_type: 1 00000080:00200000:1.0:1788572079.865364:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572079.865367:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=267 00000080:00200000:2.0:1788572079.870386:0:63057:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572079.870416:0:63057:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572079.870434:0:63057:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 48 lru_type: 1 00010000:00010000:2.0:1788572079.870452:0:63057:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 48 lru_type: 1 00000080:00200000:2.0:1788572079.870466:0:63057:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572079.870474:0:63057:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572079.870490:0:63057:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:2.0:1788572079.870497:0:63057:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a41fd7b58 00010000:00010000:2.0:1788572079.870502:0:63057:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 48 lru_type: 1 00010000:00010000:2.0:1788572079.870511:0:63057:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 49 lru_type: 1 00000080:00200000:2.0:1788572079.870526:0:63057:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572079.870534:0:63057:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=24 00000080:00200000:2.0:1788572079.870552:0:63057:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572079.870559:0:63057:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572079.870565:0:63057:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 49 lru_type: 1 00010000:00010000:2.0:1788572079.870573:0:63057:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 49 lru_type: 1 00000080:00200000:2.0:1788572079.870579:0:63057:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572079.870583:0:63057:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572079.870586:0:63057:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:2.0:1788572079.870589:0:63057:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a41fd7c10 00010000:00010000:2.0:1788572079.870593:0:63057:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 49 lru_type: 1 00010000:00010000:2.0:1788572079.870606:0:63057:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 50 lru_type: 1 00000080:00200000:2.0:1788572079.870613:0:63057:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572079.870616:0:63057:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=24 00000080:00200000:2.0:1788572079.895003:0:63058:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572079.895028:0:63058:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572079.895046:0:63058:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 50 lru_type: 1 00010000:00010000:2.0:1788572079.895061:0:63058:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 50 lru_type: 1 00000080:00200000:2.0:1788572079.895069:0:63058:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572079.895076:0:63058:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572079.895080:0:63058:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:2.0:1788572079.895087:0:63058:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a42007b58 00010000:00010000:2.0:1788572079.895091:0:63058:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 50 lru_type: 1 00010000:00010000:2.0:1788572079.895100:0:63058:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 51 lru_type: 1 00000080:00200000:2.0:1788572079.895113:0:63058:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572079.895120:0:63058:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=8 00010000:00010000:2.0:1788572080.776117:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572081.798024:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572082.821389:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572083.856414:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572084.874869:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572084.936139:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488440064:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572084.936252:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488440320:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572084.936299:0:2405:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:2.0:1788572084.936306:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488439936:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572084.936336:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488440192:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572084.936360:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488440448:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572084.940163:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488439936:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572084.940187:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488440192:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572084.940198:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488440448:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572084.940247:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488440064:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572084.940259:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488440320:192.168.202.125@tcp:400:kworker.0 00000080:00200000:2.0:1788572085.889872:0:62287:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572085.889901:0:62287:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572085.889920:0:62287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 51 lru_type: 1 00010000:00010000:2.0:1788572085.889936:0:62287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 51 lru_type: 1 00000080:00200000:2.0:1788572085.889944:0:62287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572085.889951:0:62287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572085.889955:0:62287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:2.0:1788572085.889962:0:62287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a40da3bf8 00010000:00010000:2.0:1788572085.889966:0:62287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 51 lru_type: 1 00010000:00010000:2.0:1788572085.889975:0:62287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 52 lru_type: 1 00000080:00200000:2.0:1788572085.889988:0:62287:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572085.889995:0:62287:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=267 00010000:00010000:3.0:1788572085.901884:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1788572085.904743:0:63482:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572085.904778:0:63482:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572085.904798:0:63482:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 52 lru_type: 1 00010000:00010000:2.0:1788572085.904873:0:63482:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 52 lru_type: 1 00000080:00200000:2.0:1788572085.904883:0:63482:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572085.904891:0:63482:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572085.904895:0:63482:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:2.0:1788572085.904903:0:63482:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a4209fb58 00010000:00010000:2.0:1788572085.904907:0:63482:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 52 lru_type: 1 00010000:00010000:2.0:1788572085.904916:0:63482:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 53 lru_type: 1 00000080:00200000:2.0:1788572085.904934:0:63482:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572085.904941:0:63482:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=24 00000080:00200000:2.0:1788572085.904961:0:63482:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572085.904969:0:63482:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572085.904976:0:63482:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 53 lru_type: 1 00010000:00010000:2.0:1788572085.904985:0:63482:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 53 lru_type: 1 00000080:00200000:2.0:1788572085.904993:0:63482:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572085.904997:0:63482:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572085.904999:0:63482:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:2.0:1788572085.905003:0:63482:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a4209fc10 00010000:00010000:2.0:1788572085.905006:0:63482:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 53 lru_type: 1 00010000:00010000:2.0:1788572085.905084:0:63482:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 54 lru_type: 1 00000080:00200000:2.0:1788572085.905096:0:63482:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572085.905157:0:63482:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=24 00000080:00200000:2.0:1788572085.921779:0:63483:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572085.921804:0:63483:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572085.921821:0:63483:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 54 lru_type: 1 00010000:00010000:2.0:1788572085.921842:0:63483:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 54 lru_type: 1 00000080:00200000:2.0:1788572085.921849:0:63483:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572085.921856:0:63483:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572085.921859:0:63483:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:2.0:1788572085.921866:0:63483:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a4203fb58 00010000:00010000:2.0:1788572085.921871:0:63483:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 54 lru_type: 1 00010000:00010000:2.0:1788572085.921880:0:63483:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 55 lru_type: 1 00000080:00200000:2.0:1788572085.921893:0:63483:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572085.921900:0:63483:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d120.replay-single, flags=8 00010000:00010000:3.0:1788572086.917428:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788572087.603022:0:63626:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 121: lock replay timed out and race ========================================================== 21:34:46 (1788572086) 00010000:00010000:2.0:1788572087.942372:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1788572088.851871:0:63651:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:3.0:1788572088.851904:0:63651:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572088.851924:0:63651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 55 lru_type: 1 00010000:00010000:3.0:1788572088.851942:0:63651:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 55 lru_type: 1 00000080:00200000:3.0:1788572088.851950:0:63651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572088.851960:0:63651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:3.0:1788572088.851963:0:63651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:3.0:1788572088.851972:0:63651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a4147bbf8 00010000:00010000:3.0:1788572088.851976:0:63651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 55 lru_type: 1 00010000:00010000:3.0:1788572088.851985:0:63651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 56 lru_type: 1 00000080:00200000:3.0:1788572088.852002:0:63651:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:3.0:1788572088.852017:0:63651:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:3.0:1788572088.852024:0:63651:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572088.852031:0:63651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 56 lru_type: 1 00010000:00010000:3.0:1788572088.852039:0:63651:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 56 lru_type: 1 00000080:00200000:3.0:1788572088.852045:0:63651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572088.852049:0:63651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:3.0:1788572088.852052:0:63651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0932 00000080:00010000:3.0:1788572088.852055:0:63651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0932 from it ffffb68a4147bb38 00010000:00010000:3.0:1788572088.852060:0:63651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 56 lru_type: 1 00010000:00010000:3.0:1788572088.852067:0:63651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031c00/0x1e260726fbbc0932 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df379 expref: -99 pid: 62867 timeout: 0 lvb_type: 0 lru_score: 57 lru_type: 1 00000080:00200000:3.0:1788572088.852074:0:63651:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 3 00000080:00200000:3.0:1788572088.852079:0:63651:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f121.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), file ffff94d408407e00, open_flags 8941, mode 81b6 opened 0 00000080:00200000:3.0:1788572088.852087:0:63651:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f121.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), intent=open|creat 00000100:00100000:3.0:1788572088.858137:0:63651:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job touch:9bf287c0-f582-4b83-84c1-15e766218620:63651:1875450488440576:192.168.202.125@tcp:700:touch.0 00000100:00100000:3.0:1788572088.862987:0:63651:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d41245e840 going to sleep for 11 seconds 00000080:00200000:1.0:1788572088.939923:0:63651:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a041:0x1:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1788572088.939949:0:63651:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file f121.replay-single:[0x20000a041:0x1:0x0](ffff94d410bcaa08) current size 0, valid attrs 100201f0, mode 94d4, uid 1095220816, gid -18806, new size 18446744071809989143, atime 1788572088.000000000, mtime 1788572088.000000000, ctime 1788572088.000000000, rc 0 00000080:00200008:1.0:1788572088.939977:0:63651:0:(file.c:479:ll_file_release()) START file f121.replay-single:[0x20000a041:0x1:0x0](ffff94d410bcaa08), flags 104001 00000002:00100000:1.0:1788572088.940052:0:63651:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff94d408d3f100 x1875450488440832/t90194313217(90194313217) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 592/664 e 0 to 0 dl 1788572104 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572088.940064:0:63651:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572088.940070:0:63651:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d4049be680 x1875450488441600/t0(0) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572088.940093:0:63651:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job touch:9bf287c0-f582-4b83-84c1-15e766218620:63651:1875450488441600:192.168.202.125@tcp:35:touch.0 00000100:00100000:1.0:1788572088.940177:0:63651:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4124bfcc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572088.953415:0:63651:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job touch:9bf287c0-f582-4b83-84c1-15e766218620:63651:1875450488441600:192.168.202.125@tcp:35:touch.0 00000002:00100000:1.0:1788572088.953453:0:63651:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff94d408d3f100 x1875450488440832/t90194313217(90194313217) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 592/664 e 0 to 0 dl 1788572104 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1788572088.970094:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572088.982576:0:42448:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788572088.982586:0:42448:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167800/0x1e260726fbbc094e lrc: 2/0,0 mode: CR/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xec901abfeb4df3d4 expref: -99 pid: 63651 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572088.982643:0:42448:0:(vvp_object.c:117:vvp_conf_set()) [0x20000a041:0x1:0x0]: losing layout lock 00010000:00010000:3.0:1788572088.983884:0:42448:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167800/0x1e260726fbbc094e lrc: 1/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xec901abfeb4df3d4 expref: -99 pid: 63651 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572088.983898:0:42448:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1788572088.983927:0:42448:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_01:9bf287c0-f582-4b83-84c1-15e766218620:42448:1875450488441728:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788572088.984132:0:42448:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d41245e9c0 going to sleep for 11 seconds 00010000:00010000:2.0:1788572088.985417:0:63653:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff94d405cc3800 00010000:00010000:2.0:1788572088.985470:0:63653:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 23/23 locks to regular blp list, 1 blwis in pool 00010000:00010000:3.0:1788572088.985593:0:42449:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 23 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788572088.987571:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572088.987577:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572088.987579:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572088.987612:0:42449:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d4089a3e00/0x1e260726fbbbf7b9 lrc: 2/0,0 mode: CR/CR res: [0x240006991:0x4:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4db950 expref: -99 pid: 53409 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00100000:3.0:1788572088.987693:0:42449:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff94d4125b2d80 x1875450488207360/t73014444046(73014444046) o101->lustre-MDT0001-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/608 e 0 to 0 dl 1788571830 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788572088.987723:0:42449:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788572088.987729:0:42449:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c56d80 x1875450488441984/t0(0) o35->lustre-MDT0001-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788572088.987755:0:42449:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_02:9bf287c0-f582-4b83-84c1-15e766218620:42449:1875450488441984:192.168.202.125@tcp:35:ldlm_bl.0 00000100:00100000:3.0:1788572088.987832:0:42449:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d41245e180 going to sleep for 11 seconds 00000100:00100000:2.0:1788572088.991406:0:42448:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_01:9bf287c0-f582-4b83-84c1-15e766218620:42448:1875450488441728:192.168.202.125@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788572088.991436:0:42448:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167800/0x1e260726fbbc094e lrc: 0/0,0 mode: --/CR res: [0x20000a041:0x1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xec901abfeb4df3d4 expref: -99 pid: 63651 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572088.991467:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572088.991469:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572088.991471:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572088.992594:0:42449:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_02:9bf287c0-f582-4b83-84c1-15e766218620:42449:1875450488441984:192.168.202.125@tcp:35:ldlm_bl.0 00000002:00100000:2.0:1788572088.992628:0:42449:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff94d4125b2d80 x1875450488207360/t73014444046(73014444046) o101->lustre-MDT0001-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/608 e 0 to 0 dl 1788571830 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1788572088.992680:0:42449:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d4089a3a00/0x1e260726fbbbf7ce lrc: 2/0,0 mode: PR/PR res: [0x240006991:0x4:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4db949 expref: -99 pid: 54212 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572088.992701:0:42449:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d4089ca400/0x1e260726fbbc006b lrc: 2/0,0 mode: PR/PR res: [0x2400090a1:0x16:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dd725 expref: -99 pid: 61855 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.992721:0:42449:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583a00/0x1e260726fbbc004f lrc: 2/0,0 mode: PR/PR res: [0x2400090a1:0x15:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dd6d1 expref: -99 pid: 61854 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.992734:0:42449:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d40481fe00/0x1e260726fbbc0033 lrc: 2/0,0 mode: PR/PR res: [0x2400090a1:0x14:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dd67d expref: -99 pid: 61853 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.992746:0:42449:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d40481fa00/0x1e260726fbbc0017 lrc: 2/0,0 mode: PR/PR res: [0x2400090a1:0x13:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dd629 expref: -99 pid: 61852 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.992760:0:42449:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582000/0x1e260726fbbbfffb lrc: 2/0,0 mode: PR/PR res: [0x2400090a1:0x12:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dd5d5 expref: -99 pid: 61851 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.992772:0:42449:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d41251e800/0x1e260726fbbbffdf lrc: 2/0,0 mode: PR/PR res: [0x2400090a1:0x11:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dd581 expref: -99 pid: 61850 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.992785:0:42449:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d4089a2200/0x1e260726fbbbffc3 lrc: 2/0,0 mode: PR/PR res: [0x2400090a1:0x10:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dd52d expref: -99 pid: 61849 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.992797:0:42449:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d4089a2000/0x1e260726fbbbffa7 lrc: 2/0,0 mode: PR/PR res: [0x2400090a1:0xf:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dd4d9 expref: -99 pid: 61848 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.992810:0:42449:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d41251f800/0x1e260726fbbbff8b lrc: 2/0,0 mode: PR/PR res: [0x2400090a1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dd485 expref: -99 pid: 61847 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.992823:0:42449:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d4089a3c00/0x1e260726fbbbff6f lrc: 2/0,0 mode: PR/PR res: [0x2400090a1:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dd431 expref: -99 pid: 61846 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.992835:0:42449:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d4089a2c00/0x1e260726fbbbff53 lrc: 2/0,0 mode: PR/PR res: [0x2400090a1:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dd3dd expref: -99 pid: 61845 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.992847:0:42449:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d408ddce00/0x1e260726fbbbff37 lrc: 2/0,0 mode: PR/PR res: [0x2400090a1:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dd389 expref: -99 pid: 61844 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.992860:0:42449:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d4089a3000/0x1e260726fbbbff1b lrc: 2/0,0 mode: PR/PR res: [0x2400090a1:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dd335 expref: -99 pid: 61843 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.992872:0:42449:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d40481f200/0x1e260726fbbbfeff lrc: 2/0,0 mode: PR/PR res: [0x2400090a1:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dd2e1 expref: -99 pid: 61842 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.992885:0:42449:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d40481ea00/0x1e260726fbbbfee3 lrc: 2/0,0 mode: PR/PR res: [0x2400090a1:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dd28d expref: -99 pid: 61841 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.992898:0:42449:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d40481f000/0x1e260726fbbbfec7 lrc: 2/0,0 mode: PR/PR res: [0x2400090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dd239 expref: -99 pid: 61840 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.992910:0:42449:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d408ddd000/0x1e260726fbbbfeab lrc: 2/0,0 mode: PR/PR res: [0x2400090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dd1e5 expref: -99 pid: 61839 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.992928:0:42449:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d408ddd400/0x1e260726fbbbfe8f lrc: 2/0,0 mode: PR/PR res: [0x2400090a1:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dd191 expref: -99 pid: 61838 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.992942:0:42449:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d40481f800/0x1e260726fbbbfe73 lrc: 2/0,0 mode: PR/PR res: [0x2400090a1:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dd13d expref: -99 pid: 61837 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.992956:0:42449:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d40481f600/0x1e260726fbbbfe57 lrc: 2/0,0 mode: PR/PR res: [0x2400090a1:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dd0e9 expref: -99 pid: 61836 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.992968:0:42449:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d40481f400/0x1e260726fbbbfb2b lrc: 2/0,0 mode: PR/PR res: [0x2400088d3:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dbb56 expref: -99 pid: 58159 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572088.993018:0:42449:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d4089a3e00/0x1e260726fbbbf7b9 lrc: 1/0,0 mode: --/CR res: [0x240006991:0x4:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4db950 expref: -99 pid: 53409 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572088.993026:0:42449:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d4089a3a00/0x1e260726fbbbf7ce lrc: 1/0,0 mode: --/PR res: [0x240006991:0x4:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4db949 expref: -99 pid: 54212 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572088.993033:0:42449:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d4089ca400/0x1e260726fbbc006b lrc: 1/0,0 mode: --/PR res: [0x2400090a1:0x16:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dd725 expref: -99 pid: 61855 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.993039:0:42449:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583a00/0x1e260726fbbc004f lrc: 1/0,0 mode: --/PR res: [0x2400090a1:0x15:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dd6d1 expref: -99 pid: 61854 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.993045:0:42449:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d40481fe00/0x1e260726fbbc0033 lrc: 1/0,0 mode: --/PR res: [0x2400090a1:0x14:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dd67d expref: -99 pid: 61853 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.993050:0:42449:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d40481fa00/0x1e260726fbbc0017 lrc: 1/0,0 mode: --/PR res: [0x2400090a1:0x13:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dd629 expref: -99 pid: 61852 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.993057:0:42449:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582000/0x1e260726fbbbfffb lrc: 1/0,0 mode: --/PR res: [0x2400090a1:0x12:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dd5d5 expref: -99 pid: 61851 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.993062:0:42449:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d41251e800/0x1e260726fbbbffdf lrc: 1/0,0 mode: --/PR res: [0x2400090a1:0x11:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dd581 expref: -99 pid: 61850 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.993068:0:42449:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d4089a2200/0x1e260726fbbbffc3 lrc: 1/0,0 mode: --/PR res: [0x2400090a1:0x10:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dd52d expref: -99 pid: 61849 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.993075:0:42449:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d4089a2000/0x1e260726fbbbffa7 lrc: 1/0,0 mode: --/PR res: [0x2400090a1:0xf:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dd4d9 expref: -99 pid: 61848 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.993082:0:42449:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d41251f800/0x1e260726fbbbff8b lrc: 1/0,0 mode: --/PR res: [0x2400090a1:0xe:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dd485 expref: -99 pid: 61847 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.993089:0:42449:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d4089a3c00/0x1e260726fbbbff6f lrc: 1/0,0 mode: --/PR res: [0x2400090a1:0xd:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dd431 expref: -99 pid: 61846 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.993095:0:42449:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d4089a2c00/0x1e260726fbbbff53 lrc: 1/0,0 mode: --/PR res: [0x2400090a1:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dd3dd expref: -99 pid: 61845 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.993101:0:42449:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d408ddce00/0x1e260726fbbbff37 lrc: 1/0,0 mode: --/PR res: [0x2400090a1:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dd389 expref: -99 pid: 61844 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.993107:0:42449:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d4089a3000/0x1e260726fbbbff1b lrc: 1/0,0 mode: --/PR res: [0x2400090a1:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dd335 expref: -99 pid: 61843 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.993114:0:42449:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d40481f200/0x1e260726fbbbfeff lrc: 1/0,0 mode: --/PR res: [0x2400090a1:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dd2e1 expref: -99 pid: 61842 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.993132:0:42449:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d40481ea00/0x1e260726fbbbfee3 lrc: 1/0,0 mode: --/PR res: [0x2400090a1:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dd28d expref: -99 pid: 61841 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.993138:0:42449:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d40481f000/0x1e260726fbbbfec7 lrc: 1/0,0 mode: --/PR res: [0x2400090a1:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dd239 expref: -99 pid: 61840 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.993144:0:42449:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d408ddd000/0x1e260726fbbbfeab lrc: 1/0,0 mode: --/PR res: [0x2400090a1:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dd1e5 expref: -99 pid: 61839 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.993151:0:42449:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d408ddd400/0x1e260726fbbbfe8f lrc: 1/0,0 mode: --/PR res: [0x2400090a1:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dd191 expref: -99 pid: 61838 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.993157:0:42449:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d40481f800/0x1e260726fbbbfe73 lrc: 1/0,0 mode: --/PR res: [0x2400090a1:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dd13d expref: -99 pid: 61837 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.993164:0:42449:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d40481f600/0x1e260726fbbbfe57 lrc: 1/0,0 mode: --/PR res: [0x2400090a1:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dd0e9 expref: -99 pid: 61836 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572088.993170:0:42449:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d40481f400/0x1e260726fbbbfb2b lrc: 1/0,0 mode: --/PR res: [0x2400088d3:0x3:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dbb56 expref: -99 pid: 58159 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572088.993176:0:42449:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 23 locks packed 00000100:00100000:2.0:1788572088.993580:0:42449:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_02:9bf287c0-f582-4b83-84c1-15e766218620:42449:1875450488442112:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788572088.994929:0:42449:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408d00180 going to sleep for 11 seconds 00000080:00200000:2.0:1788572089.158494:0:63659:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572089.158523:0:63659:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572089.158581:0:63659:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 63659 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572089.158595:0:63659:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 63659 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572089.158605:0:63659:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 63659 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572089.158629:0:63659:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job multiop:9bf287c0-f582-4b83-84c1-15e766218620:63659:1875450488442240:192.168.202.125@tcp:101:multiop.0 00000100:00100000:2.0:1788572089.158751:0:63659:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408d00900 going to sleep for 11 seconds 00000100:00100000:2.0:1788572089.161445:0:63659:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job multiop:9bf287c0-f582-4b83-84c1-15e766218620:63659:1875450488442240:192.168.202.125@tcp:101:multiop.0 00010000:00010000:2.0:1788572089.161471:0:63659:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df3e2 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572089.161483:0:63659:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df3e2 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572089.161494:0:63659:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df3e2 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572089.161504:0:63659:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df3e2 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572089.161516:0:63659:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df3e2 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572089.161524:0:63659:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df3e2 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572089.161537:0:63659:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df3e2 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572089.161547:0:63659:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408c60700 x1875450488442240/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572105 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572089.161560:0:63659:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df3e2 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572089.161571:0:63659:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788572089.161577:0:63659:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572089.161605:0:63659:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788572089.161610:0:63659:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0955 00000080:00010000:2.0:1788572089.161614:0:63659:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572089.161617:0:63659:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0955 from it ffffb68a41067b58 00010000:00010000:2.0:1788572089.161621:0:63659:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df3e2 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572089.161630:0:63659:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df3e2 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572089.161652:0:63659:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572089.161670:0:63659:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f121.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), flags=9 00000080:00200000:2.0:1788572089.161674:0:63659:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f121.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), intent=getattr 00000002:00010000:2.0:1788572089.161692:0:63659:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f121.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572089.161722:0:63659:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc095c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 63659 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572089.161729:0:63659:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc095c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 63659 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572089.161738:0:63659:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc095c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 63659 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572089.161755:0:63659:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job multiop:9bf287c0-f582-4b83-84c1-15e766218620:63659:1875450488442368:192.168.202.125@tcp:101:multiop.0 00000100:00100000:2.0:1788572089.161855:0:63659:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408d00900 going to sleep for 11 seconds 00000100:00100000:2.0:1788572089.176296:0:63659:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job multiop:9bf287c0-f582-4b83-84c1-15e766218620:63659:1875450488442368:192.168.202.125@tcp:101:multiop.0 00010000:00010000:2.0:1788572089.176325:0:63659:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc095c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df3fe expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572089.176348:0:63659:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc095c lrc: 4/1,0 mode: --/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df3fe expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572089.176357:0:63659:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc095c lrc: 4/1,0 mode: --/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df3fe expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572089.176368:0:63659:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc095c lrc: 4/1,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df3fe expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572089.176380:0:63659:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc095c lrc: 4/1,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df3fe expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572089.176394:0:63659:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc095c lrc: 5/2,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df3fe expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572089.176408:0:63659:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc095c lrc: 5/2,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df3fe expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572089.176416:0:63659:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc095c lrc: 4/1,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df3fe expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572089.176427:0:63659:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408c60700 x1875450488442368/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 592/640 e 0 to 0 dl 1788572105 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572089.176439:0:63659:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc095c lrc: 3/1,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df3fe expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572089.176452:0:63659:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc095c lrc: 3/1,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df3fe expref: -99 pid: 63659 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572089.176464:0:63659:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589975617/1 (0 0) 00000080:00200000:2.0:1788572089.176471:0:63659:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572089.176564:0:63659:0:(namei.c:147:ll_iget()) got inode: [0x20000a041:0x1:0x0](ffff94d410bcaa08): rc = 0 00000080:00200000:2.0:1788572089.176585:0:63659:0:(file.c:6826:ll_layout_conf()) [0x20000a041:0x1:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1788572089.176595:0:63659:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a041:0x1:0x0](ffff94d410bcaa08) for lock 0x1e260726fbbc095c 00000080:00010000:2.0:1788572089.176606:0:63659:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a041:0x1:0x0](ffff94d410bcaa08) 00000080:00010000:2.0:1788572089.176609:0:63659:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc095c from it ffffb68a41067b20 00010000:00010000:2.0:1788572089.176612:0:63659:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc095c lrc: 3/1,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4df3fe expref: -99 pid: 63659 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572089.176622:0:63659:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc095c lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4df3fe expref: -99 pid: 63659 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788572089.176667:0:63659:0:(file.c:6285:ll_getattr_dentry()) START file f121.replay-single:[0x20000a041:0x1:0x0](ffff94d410bcaa08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572089.176675:0:63659:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff94d410bcaa08),name=f121.replay-single 00000002:00010000:2.0:1788572089.176698:0:63659:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x20000a041:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572089.176707:0:63659:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc095c lrc: 2/1,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4df3fe expref: -99 pid: 63659 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572089.176719:0:63659:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc095c lrc: 2/1,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4df3fe expref: -99 pid: 63659 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572089.176725:0:63659:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572089.176730:0:63659:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a041:0x1:0x0](ffff94d410bcaa08) 00000080:00010000:2.0:1788572089.176733:0:63659:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a041:0x1:0x0](ffff94d410bcaa08) for lock 0x1e260726fbbc095c 00000080:00010000:2.0:1788572089.176736:0:63659:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc095c from it ffffb68a41067cb8 00010000:00010000:2.0:1788572089.176739:0:63659:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc095c lrc: 3/1,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4df3fe expref: -99 pid: 63659 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572089.176746:0:63659:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc095c lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4df3fe expref: -99 pid: 63659 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572089.176770:0:63659:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a041:0x1:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788572089.176780:0:63659:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000a041:0x1:0x0] 00000008:00010000:2.0:1788572089.176804:0:63659:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572089.176807:0:63659:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff94d4057bdc30, osc lock ffff94d40857ad10, flags 0x400001002 00010000:00010000:2.0:1788572089.176848:0:63659:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000047144dbc/0x1e260726fbbc0963 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x682:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 63659 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572089.176858:0:63659:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000047144dbc/0x1e260726fbbc0963 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x682:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 63659 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788572089.176904:0:63659:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1788572089.177163:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488442496:192.168.202.125@tcp:101:multiop.0 00010000:00010000:2.0:1788572089.188756:0:2405:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000047144dbc/0x1e260726fbbc0963 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x682:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 63659 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572089.188796:0:2405:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000047144dbc/0x1e260726fbbc0963 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x682:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df405 expref: -99 pid: 63659 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572089.188815:0:2405:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000047144dbc/0x1e260726fbbc0963 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x682:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df405 expref: -99 pid: 63659 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572089.188836:0:2405:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000047144dbc/0x1e260726fbbc0963 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x682:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df405 expref: -99 pid: 63659 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788572089.188856:0:2405:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000047144dbc/0x1e260726fbbc0963 lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x682:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4df405 expref: -99 pid: 63659 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788572089.188870:0:2405:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:2.0:1788572089.188890:0:2405:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000047144dbc/0x1e260726fbbc0963 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x682:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4df405 expref: -99 pid: 63659 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572089.188899:0:2405:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000047144dbc/0x1e260726fbbc0963 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x682:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4df405 expref: -99 pid: 63659 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572089.188907:0:2405:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000047144dbc/0x1e260726fbbc0963 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x682:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4df405 expref: -99 pid: 63659 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572089.188918:0:2405:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000047144dbc/0x1e260726fbbc0963 lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x682:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4df405 expref: -99 pid: 63659 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788572089.188947:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488442496:192.168.202.125@tcp:101:multiop.0 00000080:00200000:3.0:1788572089.189261:0:63659:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000a041:0x1:0x0] updating i_size 0 i_blocks 0 00000020:00010000:3.0:1788572089.189267:0:63659:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff94d4120279d0 (ffff94d405cee488/0) at cl_lock_release():245 00000020:00010000:3.0:1788572089.189271:0:63659:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d4120279d0 (ffff94d405cee488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788572089.189277:0:63659:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d4057bdc30 (ffff94d405cee1b8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1788572089.189282:0:63659:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d4120279d0 (ffff94d405cee488/0) at cl_lock_fini():72 00000020:00010000:3.0:1788572089.189285:0:63659:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d4057bdc30 (ffff94d405cee488/1) at cl_lock_fini():72 00000080:00200000:3.0:1788572089.189322:0:63659:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a041:0x1:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:1.0:1788572089.989424:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572090.053496:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405bab800 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:00080000:2.0:1788572090.053504:0:51031:0:(import.c:2101:ptlrpc_disconnect_and_idle_import()) lustre-OST0001-osc-ffff94d405cc3800: disconnect after 20s idle 00000100:00100000:2.0:1788572090.053540:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488442752:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572090.053622:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488443008:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572090.053650:0:2405:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:1.0:1788572090.054370:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488443136:192.168.202.125@tcp:9:kworker.0 00000100:00100000:2.0:1788572090.054821:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488442624:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572090.054912:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488442880:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572090.063235:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488442752:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572090.063256:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488443008:192.168.202.125@tcp:400:kworker.0 00000100:00080000:1.0:1788572090.065387:0:2402:0:(import.c:2011:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff94d408c61c00 x1875450488443136/t0(0) o9->lustre-OST0001-osc-ffff94d405cc3800@192.168.202.125@tcp:28/4 lens 224/224 e 0 to 0 dl 1788572100 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572090.065404:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab800 lustre-OST0001_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:1.0:1788572090.065409:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488443136:192.168.202.125@tcp:9:kworker.0 00000100:00100000:3.0:1788572090.078706:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488442624:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572090.089147:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488442880:192.168.202.125@tcp:400:kworker.0 00010000:00010000:2.0:1788572091.014307:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572093.063052:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572094.086361:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572095.174922:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488443264:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572095.175942:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488443520:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572095.192050:0:2404:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff94d405cc3800: committing for last_committed 81604378725 gen 1 00000100:00100000:1.0:1788572095.192066:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488443520:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572095.200263:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488443776:192.168.202.125@tcp:38:ptlrpcd_00_00.0 00000100:00080000:1.0:1788572095.204022:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572095.204036:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-19) 00000100:00100000:1.0:1788572095.204041:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488443776:192.168.202.125@tcp:38:ptlrpcd_00_00.0 00010000:00010000:2.0:1788572096.136651:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572097.166702:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572099.205363:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572100.231756:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572100.293577:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488443904:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572100.295688:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488444032:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572100.297126:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788572100.297147:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488444160:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572100.306725:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788572100.306750:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488444288:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572100.324032:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488444288:192.168.202.125@tcp:400:kworker.0 00000100:00080000:2.0:1788572100.327747:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572100.327762:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-19) 00000100:00100000:2.0:1788572100.327768:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488444032:192.168.202.125@tcp:38:kworker.0 00010000:00010000:1.0:1788572102.281454:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572103.301516:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572105.354709:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572105.413716:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488444544:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788572105.428089:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e60431de-58b9-4f69-9048-11a3d82ddcdd:2402:1875450488444928:192.168.202.125@tcp:250:ptlrpcd_00_01.0 00000100:00080000:2.0:1788572105.428228:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572105.428234:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-11) 00000100:00100000:2.0:1788572105.428238:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488444544:192.168.202.125@tcp:38:kworker.0 00000100:00080000:2.0:1788572105.439047:0:2402:0:(import.c:1132:ptlrpc_connect_interpret()) MGC192.168.202.125@tcp: connect to target with instance 0 10000000:01000000:2.0:1788572105.439066:0:2402:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:02000400:2.0:1788572105.439076:0:2402:0:(import.c:1284:ptlrpc_connect_interpret()) Evicted from MGS (at 192.168.202.125@tcp) after server handle changed from 0xec901abfeb4defeb to 0xec901abfeb4df540 00000100:00080000:2.0:1788572105.439080:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from CONNECTING to EVICTED 00000100:00080000:2.0:1788572105.439086:0:2402:0:(import.c:1674:ptlrpc_import_recovery_state_machine()) evicted from MGS@192.168.202.125@tcp; invalidating 00000100:00100000:2.0:1788572105.442955:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e60431de-58b9-4f69-9048-11a3d82ddcdd:2402:1875450488444928:192.168.202.125@tcp:250:ptlrpcd_00_01.0 00010000:00010000:2.0:1788572106.373801:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:3.0:1788572107.461633:0:42451:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:3.0:1788572107.461642:0:42451:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-cliir-ffff94d405cc3800 from 97 10000000:01000000:3.0:1788572107.461649:0:42451:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:3.0:1788572107.461703:0:42451:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d400031800/0x1e260726fbbc096a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572107.461717:0:42451:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.125@tcp lock: ffff94d400031800/0x1e260726fbbc096a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572107.461730:0:42451:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.125@tcp lock: ffff94d400031800/0x1e260726fbbc096a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572107.461764:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488445056:192.168.202.125@tcp:101:ll_cfg_requeue.0 00000100:00100000:3.0:1788572107.461860:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d41245ec00 going to sleep for 11 seconds 00000100:00100000:1.0:1788572107.483192:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488445056:192.168.202.125@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788572107.483259:0:42451:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.125@tcp lock: ffff94d400031800/0x1e260726fbbc096a lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4df612 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572107.483283:0:42451:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.125@tcp lock: ffff94d400031800/0x1e260726fbbc096a lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xec901abfeb4df612 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572107.483360:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488445184:192.168.202.125@tcp:256:ll_cfg_requeue.0 00000100:00100000:1.0:1788572107.483447:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4124bfc00 going to sleep for 11 seconds 00000100:00100000:1.0:1788572107.499101:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488445184:192.168.202.125@tcp:256:ll_cfg_requeue.0 00000100:00080000:1.0:1788572107.499170:0:42451:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff94d405bab000 already not connected (conn 52, was 0): DISCONN 00010000:00080000:1.0:1788572107.499185:0:42451:0:(ldlm_lib.c:85:import_set_conn()) imp ffff94d405bab000@lustre-MDT0000-mdc-ffff94d405cc3800: found existing conn 192.168.202.125@tcp, moved to head 00000100:00080000:1.0:1788572107.499191:0:42451:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572107.499200:0:42451:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572107.499248:0:42451:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572107.499279:0:42451:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488445312/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 10000000:01000000:1.0:1788572107.499339:0:42451:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.202.125@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:1.0:1788572107.499349:0:42451:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d400031800/0x1e260726fbbc096a lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4df612 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572107.499358:0:42451:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.125@tcp lock: ffff94d400031800/0x1e260726fbbc096a lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4df612 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572107.499368:0:42451:0:(mgc_request.c:611:do_requeue()) updating log lustre-client 10000000:01000000:1.0:1788572107.499370:0:42451:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-client-ffff94d405cc3800 from 52 10000000:01000000:1.0:1788572107.499373:0:42451:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:1.0:1788572107.499407:0:42451:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412583e00/0x1e260726fbbc0971 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572107.499416:0:42451:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.125@tcp lock: ffff94d412583e00/0x1e260726fbbc0971 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572107.499425:0:42451:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.125@tcp lock: ffff94d412583e00/0x1e260726fbbc0971 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572107.499449:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488445440:192.168.202.125@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788572107.499621:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488445312:192.168.202.125@tcp:38:ll_cfg_requeue.0 00000100:00100000:1.0:1788572107.500337:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4124bfc00 going to sleep for 11 seconds 00000100:00100000:1.0:1788572107.503311:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488445440:192.168.202.125@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788572107.503341:0:42451:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.125@tcp lock: ffff94d412583e00/0x1e260726fbbc0971 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4df619 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572107.503354:0:42451:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.125@tcp lock: ffff94d412583e00/0x1e260726fbbc0971 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xec901abfeb4df619 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572107.503781:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488445568:192.168.202.125@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788572107.503860:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4124bfc00 going to sleep for 11 seconds 00010000:00080000:1.0:1788572107.508686:0:2402:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff94d4049be300 x1875450488445312/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/416 e 0 to 0 dl 1788572127 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572107.508704:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 90194313219 gen 3 00000100:00080000:1.0:1788572107.508740:0:2402:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to target with instance 22 00000100:00080000:1.0:1788572107.508762:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:1.0:1788572107.508768:0:2402:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:1.0:1788572107.508771:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 90194313219 gen 3 00000100:00080000:1.0:1788572107.508775:0:2402:0:(recover.c:53:ptlrpc_replay_next()) import ffff94d405bab000 from lustre-MDT0000_UUID committed 90194313219 last 0 00000100:00080000:1.0:1788572107.508779:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:2.0:1788572107.509368:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488445312:192.168.202.125@tcp:38:ll_cfg_requeue.0 00010000:00080000:1.0:1788572107.509562:0:63980:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff94d405cc3800 to lustre-MDT0000_UUID@192.168.202.125@tcp 00010000:00010000:1.0:1788572107.510064:0:63980:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df3e2 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00080000:1.0:1788572107.510106:0:63980:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 1(8) 00010000:00010000:1.0:1788572107.510117:0:63980:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc095c lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4df3fe expref: -99 pid: 63659 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00080000:1.0:1788572107.510126:0:63980:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 2(8) 00000100:00100000:2.0:1788572107.510946:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488445696:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00100000:2.0:1788572107.511042:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488445824:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00100000:1.0:1788572107.515370:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488445568:192.168.202.125@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788572107.515439:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488445952:192.168.202.125@tcp:503:ll_cfg_requeue.0 00000100:00100000:1.0:1788572107.516355:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4124bfc00 going to sleep for 11 seconds 00000100:00100000:1.0:1788572107.519823:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488445952:192.168.202.125@tcp:503:ll_cfg_requeue.0 00000040:00080000:1.0:1788572107.520747:0:63981:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 64768 count 51 00000020:01000000:3.0:1788572107.521147:0:42451:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 52-51 (rc=0) 10000000:01000000:3.0:1788572107.521169:0:42451:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.202.125@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:3.0:1788572107.521191:0:42451:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412583e00/0x1e260726fbbc0971 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4df619 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572107.521234:0:42451:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.125@tcp lock: ffff94d412583e00/0x1e260726fbbc0971 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4df619 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572107.521244:0:42451:0:(mgc_request.c:611:do_requeue()) updating log params 10000000:01000000:3.0:1788572107.521247:0:42451:0:(mgc_request.c:1801:mgc_process_log()) Process log params-ffff94d405cc3800 from 7 10000000:01000000:3.0:1788572107.521250:0:42451:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:3.0:1788572107.521290:0:42451:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d400031600/0x1e260726fbbc0978 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572107.521299:0:42451:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.125@tcp lock: ffff94d400031600/0x1e260726fbbc0978 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572107.521320:0:42451:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.125@tcp lock: ffff94d400031600/0x1e260726fbbc0978 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572107.521348:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488446080:192.168.202.125@tcp:101:ll_cfg_requeue.0 00000100:00100000:3.0:1788572107.521443:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d41245e900 going to sleep for 11 seconds 00000100:00100000:3.0:1788572107.524010:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488446080:192.168.202.125@tcp:101:ll_cfg_requeue.0 00010000:00010000:3.0:1788572107.524041:0:42451:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.125@tcp lock: ffff94d400031600/0x1e260726fbbc0978 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4df620 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572107.524057:0:42451:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.125@tcp lock: ffff94d400031600/0x1e260726fbbc0978 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xec901abfeb4df620 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572107.524135:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488446208:192.168.202.125@tcp:501:ll_cfg_requeue.0 00000100:00100000:3.0:1788572107.524542:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d41245e900 going to sleep for 11 seconds 00000100:00100000:3.0:1788572107.533367:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488446208:192.168.202.125@tcp:501:ll_cfg_requeue.0 00000100:00100000:3.0:1788572107.533438:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488446336:192.168.202.125@tcp:503:ll_cfg_requeue.0 00000100:00100000:3.0:1788572107.533548:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d41245e900 going to sleep for 11 seconds 00000100:00100000:3.0:1788572107.541126:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488446336:192.168.202.125@tcp:503:ll_cfg_requeue.0 00000040:00080000:1.0:1788572107.544001:0:63982:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 64768 count 7 00010000:00010000:1.0:1788572108.421421:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572109.445556:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572110.536741:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 1 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572110.536749:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:2.0:1788572110.536795:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488446848:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572110.536898:0:2405:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:2.0:1788572110.536906:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488446720:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572110.537287:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488446976:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572110.543955:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488446720:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572110.543981:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488446976:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572110.543992:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488446848:192.168.202.125@tcp:400:kworker.0 00010000:00010000:1.0:1788572111.493480:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572112.517543:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572112.586431:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488447104:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572112.586528:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488447360:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572112.586562:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788572112.586568:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488447232:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572112.589945:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488447232:192.168.202.125@tcp:400:kworker.0 00010000:00010000:1.0:1788572114.566407:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:3.0:1788572115.369785:0:64052:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572115.369801:0:64052:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:3.0:1788572115.369826:0:64052:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572115.369843:0:64052:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df3e2 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572115.370132:0:64052:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df3e2 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572115.370141:0:64052:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572115.370151:0:64052:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:3.0:1788572115.370154:0:64052:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0955 00000080:00010000:3.0:1788572115.370162:0:64052:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0955 from it ffffb68a40d4bcb8 00010000:00010000:3.0:1788572115.370166:0:64052:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df3e2 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572115.370176:0:64052:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df3e2 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200008:3.0:1788572115.371071:0:64052:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788572115.371079:0:64052:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:3.0:1788572115.371094:0:64052:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572115.371108:0:64052:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df3e2 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572115.371119:0:64052:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df3e2 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572115.371126:0:64052:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572115.371132:0:64052:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:3.0:1788572115.371135:0:64052:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0955 00000080:00010000:3.0:1788572115.371140:0:64052:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0955 from it ffffb68a40d4bcb8 00010000:00010000:3.0:1788572115.371144:0:64052:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df3e2 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572115.371152:0:64052:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df3e2 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788572115.371313:0:64052:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:3.0:1788572115.371322:0:64052:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572115.371329:0:64052:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df3e2 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572115.371339:0:64052:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df3e2 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788572115.371345:0:64052:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572115.371348:0:64052:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:3.0:1788572115.371351:0:64052:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0955 00000080:00010000:3.0:1788572115.371354:0:64052:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0955 from it ffffb68a40d4bbb0 00010000:00010000:3.0:1788572115.371357:0:64052:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df3e2 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572115.371364:0:64052:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df3e2 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1788572115.371372:0:64052:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 44 00000080:00200008:3.0:1788572115.371379:0:64052:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 100000 00000080:00200080:3.0:1788572115.371629:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572115.371647:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00800000:00000004:3.0:1788572115.371654:0:64052:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0000-mdc-ffff94d405cc3800: age 3224, max_age 3267 00000100:00080000:3.0:1788572115.371698:0:64052:0:(client.c:1832:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != REPLAY_LOCKS) req@ffff94d408c54000 x1875450488447488/t0(0) o41->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572115.371708:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:2.0:1788572115.591101:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572116.421615:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00000100:00100000:3.0:1788572117.445359:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572117.640263:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572118.469529:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:2.0:1788572118.662869:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572119.499479:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00000080:00200000:3.0:1788572119.750587:0:36550:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a041:0x1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572119.750601:0:36550:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572119.750654:0:36550:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a041:0x1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1788572120.519599:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572120.709546:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572121.541577:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:2.0:1788572121.733477:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572122.565517:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00000100:00080000:3.0:1788572122.824716:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572122.824723:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:3.0:1788572122.824764:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488448128:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572122.824851:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788572122.824858:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488448000:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572122.824893:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488448256:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572122.827292:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488448000:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572122.829119:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488448256:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572122.841199:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488448128:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572123.593564:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572123.783273:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:1.0:1788572123.845364:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572107/real 1788572107] req@ffff94d4049be680 x1875450488445696/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 328/344 e 0 to 1 dl 1788572123 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572123.845401:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff94d4049be680 x1875450488445696/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 328/344 e 0 to 1 dl 1788572123 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000400:1.0:1788572123.845412:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572107/real 1788572107] req@ffff94d4049bdc00 x1875450488445824/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 328/400 e 0 to 1 dl 1788572123 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572123.845421:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff94d4049bdc00 x1875450488445824/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 328/400 e 0 to 1 dl 1788572123 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572123.845440:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to CONNECTING 00000100:00080000:1.0:1788572123.845445:0:2402:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3260 00000100:00080000:1.0:1788572123.845459:0:2402:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572123.845473:0:2402:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572123.845505:0:2402:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd880 x1875450488448384/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572123.845519:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488445696:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00020000:1.0:1788572123.845525:0:2402:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting 00000100:00100000:1.0:1788572123.856702:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488445824:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00100000:1.0:1788572123.856755:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488448384:192.168.202.125@tcp:38:ptlrpcd_rcv.0 00010000:00080000:1.0:1788572123.920082:0:2402:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff94d4049bd880 x1875450488448384/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/416 e 0 to 0 dl 1788572133 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572123.920108:0:2402:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to target with instance 22 00000100:00080000:1.0:1788572123.920130:0:2402:0:(import.c:1297:ptlrpc_connect_interpret()) reconnected to lustre-MDT0000_UUID@192.168.202.125@tcp after partition 00000100:00080000:1.0:1788572123.920133:0:2402:0:(import.c:1307:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff94d405cc3800: reconnected to lustre-MDT0000_UUID during replay 00000100:00080000:1.0:1788572123.920136:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:1.0:1788572123.920140:0:2402:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:1.0:1788572123.920143:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 90194313219 gen 3 00000100:00080000:1.0:1788572123.920146:0:2402:0:(recover.c:53:ptlrpc_replay_next()) import ffff94d405bab000 from lustre-MDT0000_UUID committed 90194313219 last 0 00000100:00080000:1.0:1788572123.920149:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:1.0:1788572123.920465:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488448384:192.168.202.125@tcp:38:ptlrpcd_rcv.0 00010000:00080000:3.0:1788572123.920533:0:64068:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff94d405cc3800 to lustre-MDT0000_UUID@192.168.202.125@tcp 00010000:00010000:3.0:1788572123.921310:0:64068:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df3e2 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000100:00100000:1.0:1788572123.921688:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488448512:192.168.202.125@tcp:101:ldlm_lock_repla.0 00010000:00080000:3.0:1788572123.922545:0:64068:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 1(8) 00010000:00010000:3.0:1788572123.922577:0:64068:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc095c lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4df3fe expref: -99 pid: 63659 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000100:00100000:1.0:1788572123.925172:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488448640:192.168.202.125@tcp:101:ldlm_lock_repla.0 00010000:00080000:3.0:1788572123.930087:0:64068:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 2(8) 00000100:00100000:3.0:1788572124.617734:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:2.0:1788572124.805593:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572126.856384:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572127.685514:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:2.0:1788572127.884062:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572127.941831:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572127.941839:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:1.0:1788572127.941890:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488448896:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572127.965243:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488448896:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572128.709668:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00000100:00080000:1.0:1788572128.965572:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572128.965580:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:3.0:1788572129.733794:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572129.930544:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572130.758030:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:2.0:1788572130.951244:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572131.781658:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00000100:00100000:3.0:1788572132.805895:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572132.997392:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572133.830789:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:2.0:1788572134.021410:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572134.088141:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572134.088148:0:51031:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:1.0:1788572134.088187:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488449664:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572134.088311:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:1.0:1788572134.088319:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488449536:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572134.088352:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488449792:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572134.091174:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488449536:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572134.091195:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488449792:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572134.091231:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488449664:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572134.854096:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00000100:00100000:3.0:1788572135.877317:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572136.070837:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572136.902141:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:2.0:1788572137.094322:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572137.925359:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00000100:00100000:3.0:1788572138.949377:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572139.144794:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572139.208103:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572139.208111:0:51031:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:1.0:1788572139.208145:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488450048:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572139.208253:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:1.0:1788572139.208260:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488449920:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572139.208294:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488450176:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572139.217352:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488449920:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572139.217379:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488450048:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572139.219161:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488450176:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572139.308252:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488450304:192.168.202.125@tcp:38:ptlrpcd_rcv.0 00000100:00100000:1.0:1788572139.310582:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488450432:192.168.202.125@tcp:101:ldlm_lock_repla.0 00010000:00080000:2.0:1788572139.314227:0:64072:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 1(8) 00010000:00010000:2.0:1788572139.314274:0:64072:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc095c lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4df3fe expref: -99 pid: 63659 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00080000:2.0:1788572139.314308:0:64072:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 2(8) 00000100:00100000:1.0:1788572139.314459:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488450560:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00100000:3.0:1788572139.973676:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:2.0:1788572140.172071:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572141.003150:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00000100:00100000:3.0:1788572142.021567:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572142.213422:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572143.045475:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:2.0:1788572143.237461:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572144.070018:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00000100:00080000:2.0:1788572144.325384:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572144.325390:0:51031:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:2.0:1788572144.325464:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488450688:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572144.325494:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488450816:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572144.325560:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488450944:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572144.334340:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488450688:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572144.337003:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488450816:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572144.337381:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488450944:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572145.093413:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572145.285475:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572146.117372:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:2.0:1788572146.309501:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572147.141363:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00000100:00100000:3.0:1788572148.165346:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572148.357544:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572149.189468:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:2.0:1788572149.382546:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572149.446458:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572149.446465:0:51031:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:3.0:1788572149.446578:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488451072:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572149.446669:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488451328:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572149.446703:0:2405:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788572149.446710:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488451200:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572149.451688:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488451328:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572149.452099:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488451072:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572149.457290:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488451200:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572150.213319:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00000100:00100000:3.0:1788572151.237475:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572151.429626:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572152.261441:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:2.0:1788572152.455832:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572153.285460:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572153.478379:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572154.501558:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572154.565760:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572154.565767:0:51031:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:3.0:1788572154.566186:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488451456:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572154.566313:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488451712:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572154.566346:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788572154.566352:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488451584:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572154.574463:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488451584:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572154.574482:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488451456:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572154.574493:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488451712:192.168.202.125@tcp:400:kworker.0 00010000:00010000:3.0:1788572155.528904:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572155.615880:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488451840:192.168.202.125@tcp:38:ptlrpcd_rcv.0 00000100:00100000:1.0:1788572155.616544:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488451968:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00100000:1.0:1788572155.619298:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488452096:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00100000:3.0:1788572156.359437:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572156.551217:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572157.381504:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572157.574215:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572158.405331:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572158.597484:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572159.429312:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572159.622008:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572159.685538:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572159.685544:0:51031:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:1.0:1788572159.685585:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488452352:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572159.685685:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:1.0:1788572159.685693:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488452224:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572159.685725:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488452480:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572159.688650:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488452224:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572159.688675:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488452352:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572159.692658:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488452480:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572160.453446:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572160.646854:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572160.709731:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572160.709738:0:51031:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:3.0:1788572160.709859:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488452608:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572160.709903:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488452736:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572160.709967:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488452864:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572160.715432:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488452736:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572160.715495:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488452608:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572160.715507:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488452864:192.168.202.125@tcp:400:kworker.0 00010000:00010000:3.0:1788572161.669454:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572162.502448:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572162.693992:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572163.525446:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572163.718061:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572164.549585:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572164.741384:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572165.573593:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572165.765476:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572165.829767:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572165.829775:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:3.0:1788572166.597343:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572166.790728:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572167.622273:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572167.814386:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572168.646185:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572168.842666:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572169.669345:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572169.861980:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572170.693317:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572170.885437:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572170.950137:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488453376:192.168.202.125@tcp:400:kworker.0 00000100:00080000:2.0:1788572170.950305:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572170.950311:0:51031:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:3.0:1788572170.957616:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788572170.957641:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488453504:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572170.959688:0:2405:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788572170.959709:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488453632:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572170.961720:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488453376:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572170.967998:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488453632:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572170.976997:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488453504:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572171.717418:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572171.911791:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572172.008269:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488453760:192.168.202.125@tcp:38:ptlrpcd_rcv.0 00000100:00100000:1.0:1788572172.009907:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488453888:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00100000:1.0:1788572172.010004:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488454016:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00100000:3.0:1788572172.741565:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572172.938885:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572173.765551:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572173.972088:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572174.789586:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572174.981600:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572176.005680:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572176.070392:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572176.070400:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:2.0:1788572176.070506:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488454144:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572176.070719:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488454400:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572176.070756:0:2405:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788572176.070761:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488454272:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572176.081142:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488454272:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572176.081160:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488454144:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572176.081173:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488454400:192.168.202.125@tcp:400:kworker.0 00010000:00010000:3.0:1788572177.030677:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572177.093474:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572177.093482:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:3.0:1788572177.093526:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488454656:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572177.093613:0:2405:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1788572177.093620:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488454528:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572177.093661:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488454784:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572177.097670:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488454784:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572177.100050:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488454528:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572177.103582:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488454656:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572177.863199:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572178.057009:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572178.885332:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572179.077696:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572179.909326:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572180.101645:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572180.933351:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572181.125661:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572181.957288:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572182.149426:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572182.213726:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572182.213734:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:3.0:1788572182.213770:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488455040:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572182.213800:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488454912:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572182.218309:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488455168:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572182.221076:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488455040:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572182.232435:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488454912:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572182.232456:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488455168:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572182.981569:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572183.175737:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572184.005485:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572184.197457:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572185.029678:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572185.221792:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572186.054265:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572186.250230:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572187.080721:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572187.269763:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572187.333694:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488455296:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572187.334296:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788572187.334307:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488455424:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572187.335142:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488455552:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572187.340064:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488455424:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572187.340082:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488455296:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572187.348652:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488455552:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572188.101628:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572188.293662:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572188.403697:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488455680:192.168.202.125@tcp:38:ptlrpcd_rcv.0 00000100:00100000:1.0:1788572188.404883:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488455808:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00100000:1.0:1788572188.405036:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488455936:192.168.202.125@tcp:101:ldlm_lock_repla.0 00010000:00010000:3.0:1788572189.317489:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572190.342387:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572191.173820:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572191.365642:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572192.197329:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572192.389589:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572192.453517:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488456192:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572192.453633:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:1.0:1788572192.453640:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488456064:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572192.453674:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488456320:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572193.221324:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572193.415371:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572193.477484:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488456448:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572193.489550:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488456704:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572193.489707:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488456448:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572193.490377:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788572193.490399:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488456576:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572193.499166:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488456576:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572193.499186:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488456704:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572194.247323:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572194.443706:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572195.269324:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572195.463478:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572196.293360:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572196.485710:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572197.513459:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572198.533420:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572198.603844:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488456832:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572198.603981:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488457088:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572198.604017:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788572198.604024:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488456960:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572198.611081:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488456960:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572198.611096:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488456832:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572198.611107:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488457088:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572199.365451:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572199.558105:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572200.389391:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572200.581437:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572201.413345:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00000400:00000100:2.0:1788572201.499927:0:2396:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.202.125@tcp portal 10 match 1875450488448512 offset 224 length 344: 4 00010000:00010000:3.0:1788572201.608686:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572202.439608:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572202.629670:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572203.461376:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572203.654340:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572203.718464:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572203.718472:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:1.0:1788572203.718511:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488457344:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572203.718597:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:1.0:1788572203.718605:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488457216:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572203.718636:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488457472:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572203.724028:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488457216:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572203.732991:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488457472:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572203.733012:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488457344:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572204.485367:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572204.677444:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1788572204.741441:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572188/real 1788572188] req@ffff94d408d3c700 x1875450488455808/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 328/344 e 0 to 1 dl 1788572204 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572204.741479:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff94d408d3c700 x1875450488455808/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 328/344 e 0 to 1 dl 1788572204 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000400:2.0:1788572204.741491:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572188/real 1788572188] req@ffff94d408d3df80 x1875450488455936/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 328/400 e 0 to 1 dl 1788572204 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572204.741502:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff94d408d3df80 x1875450488455936/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 328/400 e 0 to 1 dl 1788572204 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572204.741523:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to CONNECTING 00000100:00080000:2.0:1788572204.741528:0:2402:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3341 00000100:00080000:2.0:1788572204.741542:0:2402:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572204.741555:0:2402:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572204.742392:0:2402:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488457600/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572204.742749:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488455808:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00020000:2.0:1788572204.742757:0:2402:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting 00000100:00100000:2.0:1788572204.755504:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488455936:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00100000:2.0:1788572204.755550:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488457600:192.168.202.125@tcp:38:ptlrpcd_rcv.0 00000100:00080000:1.0:1788572204.770059:0:2402:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to target with instance 22 00000100:00080000:1.0:1788572204.770095:0:2402:0:(import.c:1297:ptlrpc_connect_interpret()) reconnected to lustre-MDT0000_UUID@192.168.202.125@tcp after partition 00000100:00080000:1.0:1788572204.770098:0:2402:0:(import.c:1307:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff94d405cc3800: reconnected to lustre-MDT0000_UUID during replay 00000100:00080000:1.0:1788572204.770102:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:1.0:1788572204.770118:0:2402:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:1.0:1788572204.770121:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 90194313219 gen 3 00000100:00080000:1.0:1788572204.770264:0:2402:0:(recover.c:53:ptlrpc_replay_next()) import ffff94d405bab000 from lustre-MDT0000_UUID committed 90194313219 last 0 00000100:00080000:1.0:1788572204.770269:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:2.0:1788572204.773889:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488457600:192.168.202.125@tcp:38:ptlrpcd_rcv.0 00010000:00080000:1.0:1788572204.773991:0:64085:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff94d405cc3800 to lustre-MDT0000_UUID@192.168.202.125@tcp 00010000:00010000:1.0:1788572204.774718:0:64085:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df3e2 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000100:00100000:2.0:1788572204.774798:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488457728:192.168.202.125@tcp:101:ldlm_lock_repla.0 00010000:00080000:1.0:1788572204.774950:0:64085:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 1(8) 00010000:00010000:1.0:1788572204.774964:0:64085:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc095c lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4df3fe expref: -99 pid: 63659 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000100:00100000:2.0:1788572204.775726:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488457856:192.168.202.125@tcp:101:ldlm_lock_repla.0 00010000:00080000:1.0:1788572204.776032:0:64085:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 2(8) 00000100:00100000:3.0:1788572205.510165:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572205.701632:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572206.533622:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572206.725461:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572207.558270:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572207.749476:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572208.582497:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572208.776370:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572208.837770:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572208.837779:0:51031:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:1.0:1788572208.837830:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488458112:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572208.841200:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488458112:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572209.605484:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572209.797533:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572209.861431:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572209.861438:0:51031:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:1.0:1788572209.861526:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488458368:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572209.861638:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488458624:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572209.866450:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488458624:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572209.872557:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488458368:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572210.629382:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572210.821382:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572211.653468:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572211.845492:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572212.682447:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572212.869674:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572213.704321:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572213.894359:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572214.725456:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572214.917624:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572214.981651:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572214.981657:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:1.0:1788572214.987742:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488458880:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572214.987852:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788572214.987859:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488459008:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572214.990865:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488459008:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572215.000488:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488458880:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572215.749555:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572215.941497:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572216.776064:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572216.965735:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572217.799587:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572217.991336:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572218.821482:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572219.013721:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572219.845540:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572220.037600:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572220.101561:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572220.101570:0:51031:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:3.0:1788572220.101691:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488459136:192.168.202.125@tcp:400:kworker.0 00000100:00000400:1.0:1788572220.102179:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572204/real 1788572204] req@ffff94d4049bea00 x1875450488457856/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 328/400 e 0 to 1 dl 1788572220 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572220.102232:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff94d4049bea00 x1875450488457856/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 328/400 e 0 to 1 dl 1788572220 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000400:1.0:1788572220.102244:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572204/real 1788572204] req@ffff94d4049bc380 x1875450488457728/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 328/344 e 0 to 1 dl 1788572220 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572220.102257:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff94d4049bc380 x1875450488457728/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 328/344 e 0 to 1 dl 1788572220 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572220.102272:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to CONNECTING 00000100:00080000:1.0:1788572220.102277:0:2402:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3358 00000100:00080000:1.0:1788572220.102287:0:2402:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572220.102298:0:2402:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572220.102316:0:2402:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488459264/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572220.102328:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488457856:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00020000:1.0:1788572220.102334:0:2402:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting 00000100:00100000:1.0:1788572220.102337:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488457728:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00100000:1.0:1788572220.102357:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488459264:192.168.202.125@tcp:38:ptlrpcd_rcv.0 00000100:00100000:3.0:1788572220.102635:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488459392:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572220.102681:0:2405:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788572220.102688:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488459520:192.168.202.125@tcp:400:kworker.0 00000100:00080000:1.0:1788572220.109333:0:2402:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to target with instance 22 00000100:00080000:1.0:1788572220.109374:0:2402:0:(import.c:1297:ptlrpc_connect_interpret()) reconnected to lustre-MDT0000_UUID@192.168.202.125@tcp after partition 00000100:00080000:1.0:1788572220.109377:0:2402:0:(import.c:1307:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff94d405cc3800: reconnected to lustre-MDT0000_UUID during replay 00000100:00080000:1.0:1788572220.109381:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:1.0:1788572220.109386:0:2402:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:1.0:1788572220.109389:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 90194313219 gen 3 00000100:00080000:1.0:1788572220.109393:0:2402:0:(recover.c:53:ptlrpc_replay_next()) import ffff94d405bab000 from lustre-MDT0000_UUID committed 90194313219 last 0 00000100:00080000:1.0:1788572220.109397:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00010000:00080000:1.0:1788572220.109851:0:64088:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff94d405cc3800 to lustre-MDT0000_UUID@192.168.202.125@tcp 00010000:00010000:1.0:1788572220.110389:0:64088:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df3e2 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00080000:1.0:1788572220.110413:0:64088:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 1(8) 00010000:00010000:1.0:1788572220.110422:0:64088:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc095c lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4df3fe expref: -99 pid: 63659 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00080000:1.0:1788572220.110430:0:64088:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 2(8) 00000100:00100000:3.0:1788572220.110571:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488459264:192.168.202.125@tcp:38:ptlrpcd_rcv.0 00000100:00100000:3.0:1788572220.110847:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488459648:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00100000:3.0:1788572220.110941:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488459776:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00100000:3.0:1788572220.113127:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488459520:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572220.113146:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488459392:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572220.113432:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488459136:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572220.869554:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572221.063747:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572221.893536:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572222.085446:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572222.918249:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572223.109457:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572223.941352:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572224.133695:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572224.965341:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572225.157858:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572225.221429:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572225.221439:0:51031:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:2.0:1788572225.221492:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488460032:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572225.221595:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:2.0:1788572225.221602:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488459904:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572225.221714:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488460160:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572225.243010:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488459904:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572225.243036:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488460160:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572225.243046:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488460032:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572225.989551:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572226.182872:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572227.013371:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572227.208342:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572228.037412:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572228.230152:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572229.061374:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572229.255225:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572230.085367:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572230.279355:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572230.345504:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572230.345512:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:1.0:1788572230.345566:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488460416:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572230.345677:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:1.0:1788572230.346570:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488460288:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572230.346635:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488460544:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572230.349199:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488460416:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572230.350005:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488460288:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572230.354634:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488460544:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572231.109664:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572231.303016:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788572231.749285:0:2396:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.202.125@tcp portal 10 match 1875450488448640 offset 224 length 344: 4 00000100:00100000:3.0:1788572232.134452:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572232.325505:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572233.157436:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572233.349524:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572234.181766:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572234.377176:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572235.205786:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572235.401296:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572235.463725:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572235.463733:0:51031:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:3.0:1788572235.464089:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488460672:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572235.464198:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488460928:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572235.464278:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788572235.464286:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488460800:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572235.471651:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488460800:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572235.484496:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488460672:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572235.485120:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488460928:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572236.229346:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572236.428527:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:1.0:1788572236.486061:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572220/real 1788572220] req@ffff94d4049bc380 x1875450488459648/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 328/344 e 0 to 1 dl 1788572236 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572236.486097:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff94d4049bc380 x1875450488459648/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 328/344 e 0 to 1 dl 1788572236 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000400:1.0:1788572236.486108:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572220/real 1788572220] req@ffff94d4049bea00 x1875450488459776/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 328/400 e 0 to 1 dl 1788572236 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572236.486120:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff94d4049bea00 x1875450488459776/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 328/400 e 0 to 1 dl 1788572236 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572236.486139:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to CONNECTING 00000100:00080000:1.0:1788572236.486144:0:2402:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3373 00000100:00080000:1.0:1788572236.486158:0:2402:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572236.486170:0:2402:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572236.486221:0:2402:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdf80 x1875450488461056/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572236.486236:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488459648:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00020000:1.0:1788572236.486242:0:2402:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting 00000100:00100000:1.0:1788572236.514423:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488459776:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00100000:1.0:1788572236.514472:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488461056:192.168.202.125@tcp:38:ptlrpcd_rcv.0 00000100:00080000:1.0:1788572236.557367:0:2402:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to target with instance 22 00000100:00080000:1.0:1788572236.557400:0:2402:0:(import.c:1297:ptlrpc_connect_interpret()) reconnected to lustre-MDT0000_UUID@192.168.202.125@tcp after partition 00000100:00080000:1.0:1788572236.557403:0:2402:0:(import.c:1307:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff94d405cc3800: reconnected to lustre-MDT0000_UUID during replay 00000100:00080000:1.0:1788572236.557406:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:1.0:1788572236.557412:0:2402:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:1.0:1788572236.557414:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 90194313219 gen 3 00000100:00080000:1.0:1788572236.557419:0:2402:0:(recover.c:53:ptlrpc_replay_next()) import ffff94d405bab000 from lustre-MDT0000_UUID committed 90194313219 last 0 00000100:00080000:1.0:1788572236.557423:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:1.0:1788572236.557939:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488461056:192.168.202.125@tcp:38:ptlrpcd_rcv.0 00010000:00080000:3.0:1788572236.558064:0:64091:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff94d405cc3800 to lustre-MDT0000_UUID@192.168.202.125@tcp 00010000:00010000:3.0:1788572236.558576:0:64091:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df3e2 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00080000:3.0:1788572236.558603:0:64091:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 1(8) 00010000:00010000:3.0:1788572236.558612:0:64091:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc095c lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4df3fe expref: -99 pid: 63659 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00080000:3.0:1788572236.558620:0:64091:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 2(8) 00000100:00100000:1.0:1788572236.558954:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488461184:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00100000:1.0:1788572236.559065:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488461312:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00100000:3.0:1788572237.253338:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572237.445477:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572238.277404:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572238.469445:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572239.301331:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572239.493715:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572240.330176:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572240.517449:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572240.581870:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572240.581878:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:2.0:1788572240.582021:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488461440:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572240.582227:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488461696:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572240.582259:0:2405:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788572240.582265:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488461568:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572240.584543:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488461568:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572240.588276:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488461440:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572240.594351:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488461696:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572241.349332:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572241.541421:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572242.373607:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572242.565611:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572243.398520:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572243.589651:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572244.421440:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572244.613932:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572245.445317:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572245.637567:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572245.702229:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572245.702233:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:1.0:1788572245.702472:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788572245.702482:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488462080:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572245.712070:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488462080:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572246.469438:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572246.665568:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572247.493341:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572247.686168:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572248.517395:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572248.711136:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572249.541853:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572249.733444:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572250.757470:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572251.782152:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572252.613756:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572252.812736:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:1.0:1788572252.869393:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572236/real 1788572236] req@ffff94d408c54a80 x1875450488461184/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 328/344 e 0 to 1 dl 1788572252 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572252.869431:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff94d408c54a80 x1875450488461184/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 328/344 e 0 to 1 dl 1788572252 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000400:1.0:1788572252.869442:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572236/real 1788572236] req@ffff94d408c57800 x1875450488461312/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 328/400 e 0 to 1 dl 1788572252 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572252.869455:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff94d408c57800 x1875450488461312/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 328/400 e 0 to 1 dl 1788572252 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572252.869478:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to CONNECTING 00000100:00080000:1.0:1788572252.869486:0:2402:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3389 00000100:00080000:1.0:1788572252.869500:0:2402:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572252.869513:0:2402:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572252.869546:0:2402:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488462592/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572252.869573:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488461184:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00020000:1.0:1788572252.869580:0:2402:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting 00000100:00100000:1.0:1788572252.869582:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488461312:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00100000:1.0:1788572252.869607:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488462592:192.168.202.125@tcp:38:ptlrpcd_rcv.0 00000100:00080000:1.0:1788572252.890033:0:2402:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to target with instance 22 00000100:00080000:1.0:1788572252.890091:0:2402:0:(import.c:1297:ptlrpc_connect_interpret()) reconnected to lustre-MDT0000_UUID@192.168.202.125@tcp after partition 00000100:00080000:1.0:1788572252.890094:0:2402:0:(import.c:1307:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff94d405cc3800: reconnected to lustre-MDT0000_UUID during replay 00000100:00080000:1.0:1788572252.890100:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:1.0:1788572252.890108:0:2402:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:1.0:1788572252.890112:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 90194313219 gen 3 00000100:00080000:1.0:1788572252.890118:0:2402:0:(recover.c:53:ptlrpc_replay_next()) import ffff94d405bab000 from lustre-MDT0000_UUID committed 90194313219 last 0 00000100:00080000:1.0:1788572252.890122:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:1.0:1788572252.890687:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488462592:192.168.202.125@tcp:38:ptlrpcd_rcv.0 00010000:00080000:3.0:1788572252.890770:0:64094:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff94d405cc3800 to lustre-MDT0000_UUID@192.168.202.125@tcp 00010000:00010000:3.0:1788572252.892450:0:64094:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df3e2 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000100:00100000:1.0:1788572252.892571:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488462720:192.168.202.125@tcp:101:ldlm_lock_repla.0 00010000:00080000:3.0:1788572252.892717:0:64094:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 1(8) 00010000:00010000:3.0:1788572252.892733:0:64094:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc095c lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4df3fe expref: -99 pid: 63659 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000100:00100000:1.0:1788572252.905187:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488462848:192.168.202.125@tcp:101:ldlm_lock_repla.0 00010000:00080000:3.0:1788572252.908414:0:64094:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 2(8) 00000100:00100000:3.0:1788572253.637647:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572253.829664:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572254.661334:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572254.858198:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572255.685535:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572255.878705:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572255.941989:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572255.941996:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:1.0:1788572255.942042:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488463104:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572255.942151:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:1.0:1788572255.942159:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488462976:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572255.942249:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488463232:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572255.960725:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488463232:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572255.960750:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488463104:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572255.977794:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488462976:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572256.712225:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572256.901717:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572257.735851:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572257.926833:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572257.989518:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572257.989526:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:1.0:1788572257.989605:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488463488:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572257.989697:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:1.0:1788572257.989704:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488463360:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572257.989736:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488463616:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572258.002437:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488463360:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572258.002462:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488463616:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572258.002472:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488463488:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572258.758217:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572258.949439:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572259.781369:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572259.974525:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572260.805924:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572261.001146:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788572261.939833:0:2397:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.202.125@tcp portal 10 match 1875450488450432 offset 224 length 344: 4 00010000:00010000:1.0:1788572262.021499:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572263.047234:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572263.110509:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572263.110516:0:51031:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:1.0:1788572263.110575:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488463872:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572263.110676:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:1.0:1788572263.110683:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488463744:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572263.110713:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488464000:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572263.119473:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488463744:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572263.119497:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488464000:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572263.119510:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488463872:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572263.884472:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572264.069499:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572264.901417:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572265.093556:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572265.925328:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572266.121139:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572266.949371:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572267.144772:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572267.974236:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572268.165704:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572268.229561:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572268.229569:0:51031:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:1.0:1788572268.229799:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488464256:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572268.229922:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:1.0:1788572268.229945:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488464128:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572268.229984:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488464384:192.168.202.125@tcp:400:kworker.0 00000100:00000400:3.0:1788572268.232010:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572252/real 1788572252] req@ffff94d408c55500 x1875450488462848/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 328/400 e 0 to 1 dl 1788572268 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572268.232043:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff94d408c55500 x1875450488462848/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 328/400 e 0 to 1 dl 1788572268 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000400:3.0:1788572268.232055:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572252/real 1788572252] req@ffff94d408c56a00 x1875450488462720/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 328/344 e 0 to 1 dl 1788572268 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572268.232064:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff94d408c56a00 x1875450488462720/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 328/344 e 0 to 1 dl 1788572268 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572268.232084:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to CONNECTING 00000100:00080000:3.0:1788572268.232091:0:2402:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3406 00000100:00080000:3.0:1788572268.232105:0:2402:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572268.232118:0:2402:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572268.232150:0:2402:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488464512/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572268.232167:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488462848:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00020000:3.0:1788572268.232174:0:2402:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting 00000100:00100000:3.0:1788572268.232176:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488462720:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00100000:3.0:1788572268.232233:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488464512:192.168.202.125@tcp:38:ptlrpcd_rcv.0 00000100:00100000:1.0:1788572268.239557:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488464256:192.168.202.125@tcp:400:kworker.0 00000100:00080000:3.0:1788572268.247735:0:2402:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to target with instance 22 00000100:00080000:3.0:1788572268.247764:0:2402:0:(import.c:1297:ptlrpc_connect_interpret()) reconnected to lustre-MDT0000_UUID@192.168.202.125@tcp after partition 00000100:00080000:3.0:1788572268.247766:0:2402:0:(import.c:1307:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff94d405cc3800: reconnected to lustre-MDT0000_UUID during replay 00000100:00080000:3.0:1788572268.247769:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:3.0:1788572268.247774:0:2402:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788572268.247776:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 90194313219 gen 3 00000100:00080000:3.0:1788572268.247780:0:2402:0:(recover.c:53:ptlrpc_replay_next()) import ffff94d405bab000 from lustre-MDT0000_UUID committed 90194313219 last 0 00000100:00080000:3.0:1788572268.247783:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:3.0:1788572268.249631:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488464512:192.168.202.125@tcp:38:ptlrpcd_rcv.0 00000100:00100000:3.0:1788572268.251254:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488464640:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00100000:3.0:1788572268.251357:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488464768:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00100000:3.0:1788572268.998677:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572269.189556:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572270.023510:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572270.218721:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572271.045314:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572271.238136:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572272.070634:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572272.261548:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572273.094165:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572273.286029:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572273.349346:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572273.349355:0:51031:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:1.0:1788572273.349405:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488465024:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572273.353007:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488465024:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572274.118650:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572274.314826:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572275.141539:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572275.333547:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572276.165349:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572276.357976:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572277.189527:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572277.386911:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572278.213327:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572278.406057:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572278.469581:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572278.469589:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:2.0:1788572278.469633:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488465408:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572278.469723:0:2405:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:2.0:1788572278.469730:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488465280:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572278.469768:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488465536:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572278.472971:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488465280:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572278.481464:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488465536:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572278.489123:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488465408:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572279.237697:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572279.430329:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572280.261356:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572280.453557:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572281.285329:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572281.479969:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572282.309362:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572282.501597:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572283.333307:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572283.525811:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572283.590145:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488465664:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572283.595949:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488465920:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572283.599305:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488465664:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572283.602779:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488465920:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572284.357304:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572284.549504:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:3.0:1788572284.613361:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572268/real 1788572268] req@ffff94d408d3f480 x1875450488464640/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 328/344 e 0 to 1 dl 1788572284 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572284.613400:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff94d408d3f480 x1875450488464640/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 328/344 e 0 to 1 dl 1788572284 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00000400:3.0:1788572284.613422:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572268/real 1788572268] req@ffff94d408d3d880 x1875450488464768/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 328/400 e 0 to 1 dl 1788572284 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572284.613435:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff94d408d3d880 x1875450488464768/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 328/400 e 0 to 1 dl 1788572284 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572284.613454:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to CONNECTING 00000100:00080000:3.0:1788572284.613460:0:2402:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3421 00000100:00080000:3.0:1788572284.613474:0:2402:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572284.613488:0:2402:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572284.613521:0:2402:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488466048/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572284.613543:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488464640:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00020000:3.0:1788572284.613549:0:2402:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting 00000100:00100000:3.0:1788572284.632544:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488464768:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00100000:3.0:1788572284.632591:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488466048:192.168.202.125@tcp:38:ptlrpcd_rcv.0 00000100:00080000:3.0:1788572284.750270:0:2402:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to target with instance 22 00000100:00080000:3.0:1788572284.750302:0:2402:0:(import.c:1297:ptlrpc_connect_interpret()) reconnected to lustre-MDT0000_UUID@192.168.202.125@tcp after partition 00000100:00080000:3.0:1788572284.750305:0:2402:0:(import.c:1307:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff94d405cc3800: reconnected to lustre-MDT0000_UUID during replay 00000100:00080000:3.0:1788572284.750309:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:3.0:1788572284.750314:0:2402:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788572284.750317:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 90194313219 gen 3 00000100:00080000:3.0:1788572284.750321:0:2402:0:(recover.c:53:ptlrpc_replay_next()) import ffff94d405bab000 from lustre-MDT0000_UUID committed 90194313219 last 0 00000100:00080000:3.0:1788572284.750325:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:1.0:1788572284.750936:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488466048:192.168.202.125@tcp:38:ptlrpcd_rcv.0 00010000:00080000:3.0:1788572284.751077:0:64101:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff94d405cc3800 to lustre-MDT0000_UUID@192.168.202.125@tcp 00010000:00010000:3.0:1788572284.751716:0:64101:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df3e2 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00080000:3.0:1788572284.751743:0:64101:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 1(8) 00010000:00010000:3.0:1788572284.751754:0:64101:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc095c lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4df3fe expref: -99 pid: 63659 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00080000:3.0:1788572284.751762:0:64101:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 2(8) 00000100:00100000:1.0:1788572284.751798:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488466176:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00100000:1.0:1788572284.751887:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488466304:192.168.202.125@tcp:101:ldlm_lock_repla.0 00010000:00010000:3.0:1788572285.573506:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572286.598700:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572287.429535:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572287.621591:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572288.453519:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572288.646008:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572288.710489:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488466432:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572288.710614:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488466688:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572288.710650:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788572288.710657:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488466560:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572288.714810:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488466432:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572288.732188:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488466560:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572288.732679:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488466688:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572289.477395:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572289.669483:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572289.738850:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488466944:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572289.738870:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488466816:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572289.739485:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488467072:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572290.501488:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572290.693998:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572291.526100:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:3.0:1788572291.717451:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788572292.133874:0:2396:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.202.125@tcp portal 10 match 1875450488450560 offset 224 length 344: 4 00000400:00000100:3.0:1788572292.227807:0:2397:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.202.125@tcp portal 10 match 1875450488451968 offset 224 length 224: 4 00000400:00000100:3.0:1788572292.227832:0:2397:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.202.125@tcp portal 10 match 1875450488452096 offset 224 length 224: 4 00000400:00000100:3.0:1788572292.227842:0:2397:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.202.125@tcp portal 10 match 1875450488453888 offset 224 length 224: 4 00000400:00000100:3.0:1788572292.227853:0:2397:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.202.125@tcp portal 10 match 1875450488454016 offset 224 length 224: 4 00000400:00000100:3.0:1788572292.227870:0:2397:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.202.125@tcp portal 10 match 1875450488455808 offset 224 length 224: 4 00000400:00000100:3.0:1788572292.227881:0:2397:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.202.125@tcp portal 10 match 1875450488455936 offset 224 length 224: 4 00000400:00000100:3.0:1788572292.227891:0:2397:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.202.125@tcp portal 10 match 1875450488457856 offset 224 length 224: 4 00000400:00000100:3.0:1788572292.227899:0:2397:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.202.125@tcp portal 10 match 1875450488457728 offset 224 length 224: 4 00000400:00000100:3.0:1788572292.227908:0:2397:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.202.125@tcp portal 10 match 1875450488459648 offset 224 length 224: 4 00000400:00000100:3.0:1788572292.227917:0:2397:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.202.125@tcp portal 10 match 1875450488459776 offset 224 length 224: 4 00000400:00000100:3.0:1788572292.227926:0:2397:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.202.125@tcp portal 10 match 1875450488461184 offset 224 length 224: 4 00000400:00000100:2.0:1788572292.229805:0:2396:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.202.125@tcp portal 10 match 1875450488461312 offset 224 length 224: 4 00000400:00000100:2.0:1788572292.229826:0:2396:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.202.125@tcp portal 10 match 1875450488462720 offset 224 length 224: 4 00000400:00000100:2.0:1788572292.229834:0:2396:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.202.125@tcp portal 10 match 1875450488462848 offset 224 length 224: 4 00000400:00000100:2.0:1788572292.229842:0:2396:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.202.125@tcp portal 10 match 1875450488464640 offset 224 length 224: 4 00000400:00000100:2.0:1788572292.229850:0:2396:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.202.125@tcp portal 10 match 1875450488464768 offset 224 length 224: 4 00000100:02020000:1.0:1788572292.230125:0:2402:0:(client.c:1492:ptlrpc_check_status()) lustre-MDT0000-mdc-ffff94d405cc3800: operation ldlm_enqueue to node 192.168.202.125@tcp failed: rc = -107 00000100:00080000:1.0:1788572292.230135:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to CONNECTING 00000100:00080000:1.0:1788572292.230149:0:2402:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3438 00000100:00080000:1.0:1788572292.230159:0:2402:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572292.230170:0:2402:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572292.230245:0:2402:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488467200/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572292.230261:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488466176:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:02020000:1.0:1788572292.230274:0:2402:0:(client.c:1492:ptlrpc_check_status()) lustre-MDT0000-mdc-ffff94d405cc3800: operation ldlm_enqueue to node 192.168.202.125@tcp failed: rc = -107 00000100:00020000:1.0:1788572292.230286:0:2402:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting 00000100:00100000:1.0:1788572292.230288:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488466304:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00100000:1.0:1788572292.230315:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488467200:192.168.202.125@tcp:38:ptlrpcd_rcv.0 00000100:00080000:1.0:1788572292.236935:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572292.236972:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-16) 00000100:00100000:1.0:1788572292.236978:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488467200:192.168.202.125@tcp:38:ptlrpcd_rcv.0 00000100:00100000:1.0:1788572292.238169:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488467328:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572292.262913:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488467584:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572292.267330:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488467328:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572292.267342:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488467584:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572292.552134:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572292.744266:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572293.573350:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572293.767965:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572294.789472:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572295.640094:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572295.813635:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572296.645445:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 0 seconds 00010000:00010000:1.0:1788572296.837542:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572297.413533:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572297.413541:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572297.413545:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572297.413559:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572297.413570:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572297.413589:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55500 x1875450488467840/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572297.413654:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488467968:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572297.413737:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788572297.413743:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488467712:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572297.413771:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488468096:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572297.413862:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488467840:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788572297.422855:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488467712:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572297.422872:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488468096:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572297.423772:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488467968:192.168.202.125@tcp:400:kworker.0 00000100:00080000:1.0:1788572297.427689:0:2402:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to target with instance 22 00000100:00080000:1.0:1788572297.427725:0:2402:0:(import.c:1338:ptlrpc_connect_interpret()) @@@ lustre-MDT0000-mdc-ffff94d405cc3800: evicting (reconnect/recover flags not set: 4) req@ffff94d408c55500 x1875450488467840/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/416 e 0 to 0 dl 1788572307 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572297.427736:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to EVICTED 00000100:02020000:1.0:1788572297.427741:0:2402:0:(import.c:1663:ptlrpc_import_recovery_state_machine()) lustre-MDT0000-mdc-ffff94d405cc3800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. 00000100:00080000:1.0:1788572297.446794:0:2402:0:(import.c:1674:ptlrpc_import_recovery_state_machine()) evicted from lustre-MDT0000_UUID@192.168.202.125@tcp; invalidating 00000100:00100000:3.0:1788572297.449811:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488467840:192.168.202.125@tcp:38:kworker.0 00000100:00080000:1.0:1788572297.449907:0:64104:0:(import.c:1598:ptlrpc_invalidate_import_thread()) thread invalidate import lustre-MDT0000-mdc-ffff94d405cc3800 to lustre-MDT0000_UUID@192.168.202.125@tcp 00000100:00080000:1.0:1788572297.449911:0:64104:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:1.0:1788572297.449919:0:64104:0:(client.c:3588:ptlrpc_abort_inflight()) @@@ aborting waiting req req@ffff94d408c54000 x1875450488447488/t0(0) o41->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572297.449940:0:64104:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 90194313219 gen 4 00000100:00100000:1.0:1788572297.449956:0:64104:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000100:00100000:3.0:1788572297.450496:0:64052:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:64052:1875450488447488:192.168.202.125@tcp:41:lfs.0 00000080:00200080:3.0:1788572297.450625:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.450641:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00800000:00000004:3.0:1788572297.450648:0:64052:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0001-mdc-ffff94d405cc3800: age 3224, max_age 3449 00000100:00100000:3.0:1788572297.450680:0:64052:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:64052:1875450488468224:192.168.202.125@tcp:41:lfs.0 00000100:00100000:3.0:1788572297.450754:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 11 seconds 00010000:00010000:1.0:1788572297.454870:0:64104:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc095c lrc: 3/0,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x12e400000000 nid: local remote: 0xec901abfeb4df3fe expref: -99 pid: 63659 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572297.454951:0:64104:0:(vvp_object.c:117:vvp_conf_set()) [0x20000a041:0x1:0x0]: losing layout lock 00010000:00010000:1.0:1788572297.455008:0:64104:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572297.455021:0:64104:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc095c lrc: 0/0,0 mode: --/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x492f400000000 nid: local remote: 0xec901abfeb4df3fe expref: -99 pid: 63659 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572297.455107:0:64104:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10e400000000 nid: local remote: 0xec901abfeb4df3e2 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572297.455120:0:64104:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572297.455126:0:64104:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536200/0x1e260726fbbc0955 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x490f400000000 nid: local remote: 0xec901abfeb4df3e2 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000100:00100000:3.0:1788572297.457059:0:64052:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:64052:1875450488468224:192.168.202.125@tcp:41:lfs.0 00800000:00000004:3.0:1788572297.457093:0:64052:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0001-mdc-ffff94d405cc3800: update ffff94d407db9e10 cache blocks 317207/353529 objects 1023684/1024000 00000080:00200080:3.0:1788572297.457821:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.457846:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.457857:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.457861:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.457866:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.457870:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.457875:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.457879:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.457883:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.457888:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.457892:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.457895:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.457899:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.457914:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.457920:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.457923:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.457927:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.457931:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.457935:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.457939:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.457944:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.457947:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.457952:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.457955:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.457958:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.457974:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.457979:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.457982:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.457986:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.457990:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.457994:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.457998:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458002:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458005:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458010:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458013:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458017:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458020:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458024:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458027:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458031:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458034:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458038:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458041:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458044:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458047:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458051:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458054:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458058:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458062:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458066:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458070:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458073:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458077:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458081:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458084:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458089:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458092:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458096:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458099:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458104:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458107:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458111:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458114:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458118:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458122:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458126:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458129:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458133:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458137:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458141:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458145:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458149:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458153:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458157:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458160:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458164:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458167:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458171:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458175:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458575:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458587:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458593:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458597:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458600:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458604:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458607:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458610:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458614:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458617:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458621:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458624:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458629:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458632:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458636:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458640:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458644:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458647:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458652:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458655:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458660:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458663:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458668:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458672:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458676:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458680:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458685:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458690:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458694:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458699:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458704:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458708:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458712:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458715:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458719:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458722:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458727:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458730:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458733:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458736:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458740:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458743:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458747:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458750:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458754:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00800000:00000080:3.0:1788572297.458757:0:64052:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00000080:00200080:3.0:1788572297.458761:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:3.0:1788572297.458768:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32800 uarg=0000000000000000 00020000:00000004:3.0:1788572297.458777:0:64052:0:(obd_class.h:1091:obd_statfs()) lustre-OST0000-osc-ffff94d405cc3800: age 3224, max_age 3449 00000100:00100000:3.0:1788572297.458823:0:64052:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:64052:1875450488468352:192.168.202.125@tcp:13:lfs.0 00000100:00100000:3.0:1788572297.458909:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69d80 going to sleep for 11 seconds 00000100:02000000:1.0:1788572297.459783:0:64104:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) lustre-MDT0000-mdc-ffff94d405cc3800: Connection restored to 192.168.202.125@tcp (at 192.168.202.125@tcp) 00000100:00100000:1.0:1788572297.468099:0:64052:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:64052:1875450488468352:192.168.202.125@tcp:13:lfs.0 00020000:00000004:1.0:1788572297.468129:0:64052:0:(obd_class.h:1117:obd_statfs()) lustre-OST0000-osc-ffff94d405cc3800: update ffff94d407dbd190 cache blocks 901369/958279 objects 261732/262144 00000080:00200080:1.0:1788572297.474648:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.474677:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00020000:00000004:1.0:1788572297.474688:0:64052:0:(obd_class.h:1091:obd_statfs()) lustre-OST0001-osc-ffff94d405cc3800: age 3224, max_age 3449 00000100:00080000:1.0:1788572297.474705:0:64052:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3222 00000100:00080000:1.0:1788572297.474719:0:64052:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d405cc3800: import ffff94d405bab800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572297.474730:0:64052:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8437760, lost: 0. 00000100:00100000:1.0:1788572297.474760:0:64052:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488468480/t0(0) o8->lustre-OST0001-osc-ffff94d405cc3800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572297.474834:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488468480:192.168.202.125@tcp:8:lfs.0 00000100:00080000:1.0:1788572297.475033:0:64052:0:(client.c:1832:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff94d4049bd180 x1875450488468608/t0(0) o13->lustre-OST0001-osc-ffff94d405cc3800@192.168.202.125@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572297.475044:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4124bf600 going to sleep for 0 seconds 00000100:00100000:1.0:1788572297.480729:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-OST0001-osc-ffff94d405cc3800: committing for last_committed 0 gen 14 00000100:00080000:1.0:1788572297.480756:0:2402:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff94d405cc3800: connect to target with instance 1 00000100:00080000:1.0:1788572297.480771:0:2402:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff94d405cc3800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:1.0:1788572297.480791:0:2402:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:1.0:1788572297.480797:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab800 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00000080:00000004:1.0:1788572297.480813:0:2402:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:1.0:1788572297.480819:0:2402:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:1.0:1788572297.480841:0:2402:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000238a6a8c): req@ffff94d4049bd180 x1875450488468608/t0(0) o13->lustre-OST0001-osc-ffff94d405cc3800@192.168.202.125@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572297.481363:0:64052:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4124bf600 going to sleep for 11 seconds 00000100:00100000:1.0:1788572297.481486:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488468480:192.168.202.125@tcp:8:lfs.0 00000100:00100000:1.0:1788572297.490949:0:64052:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:64052:1875450488468608:192.168.202.125@tcp:13:lfs.0 00020000:00000004:1.0:1788572297.490983:0:64052:0:(obd_class.h:1117:obd_statfs()) lustre-OST0001-osc-ffff94d405cc3800: update ffff94d407dbb7d0 cache blocks 901369/958279 objects 261733/262144 00000080:00200080:1.0:1788572297.491059:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491077:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491086:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491091:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491096:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491100:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491104:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491108:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491112:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491115:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491119:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491123:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491128:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491132:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491136:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491140:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491144:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491148:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491152:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491156:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491160:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491165:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491169:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491173:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491178:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491182:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491186:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491191:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491195:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491199:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491722:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491730:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491734:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491738:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491741:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491745:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491748:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491752:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491756:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491759:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491763:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491766:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491770:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491773:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491777:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491780:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491784:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491787:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491791:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491794:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491798:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491802:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491805:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491809:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491813:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491816:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491820:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491823:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491827:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491830:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491834:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491837:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491841:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491844:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491848:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491852:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491856:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491860:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491864:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491867:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491870:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491873:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491877:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491880:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491884:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491887:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.491891:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.491895:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.492032:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.492041:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.492048:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.492052:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.492056:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.492060:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.492064:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.492068:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.492072:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.492076:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.492080:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.492084:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.492087:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.492090:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.492094:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.492097:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.492100:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.492103:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.492107:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.492110:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.492114:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.492117:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.492120:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.492124:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.492128:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.492131:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.492135:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.492138:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.492141:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.492145:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.492149:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.492152:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.492156:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.492160:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.492164:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.492168:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.492171:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.492174:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.492178:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.492181:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.492184:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.492187:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.492190:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.492193:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.492197:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.492277:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572297.492285:0:64052:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff354a0f40 00020000:00000080:1.0:1788572297.492291:0:64052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200008:1.0:1788572297.492305:0:64052:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 100000 00010000:00010000:3.0:1788572297.861751:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572298.891468:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:2.0:1788572299.485975:0:64143:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572299.485993:0:64143:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572299.486025:0:64143:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572299.486097:0:64143:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0986 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 64143 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572299.486107:0:64143:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0986 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 64143 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572299.486119:0:64143:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0986 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 64143 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572299.486146:0:64143:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:64143:1875450488468736:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572299.489940:0:64143:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408d003c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572299.543112:0:64143:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:64143:1875450488468736:192.168.202.125@tcp:101:lfs.0 00010000:00010000:2.0:1788572299.543142:0:64143:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0986 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df826 expref: -99 pid: 64143 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572299.543155:0:64143:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0986 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df826 expref: -99 pid: 64143 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572299.543167:0:64143:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0986 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df826 expref: -99 pid: 64143 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572299.543177:0:64143:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0986 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df826 expref: -99 pid: 64143 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572299.543190:0:64143:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0986 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df826 expref: -99 pid: 64143 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572299.543199:0:64143:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0986 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df826 expref: -99 pid: 64143 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572299.543257:0:64143:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0986 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df826 expref: -99 pid: 64143 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572299.543274:0:64143:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408c61880 x1875450488468736/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572315 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572299.543287:0:64143:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0986 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df826 expref: -99 pid: 64143 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572299.543301:0:64143:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788572299.543308:0:64143:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572299.543341:0:64143:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788572299.543347:0:64143:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0986 00000080:00010000:2.0:1788572299.543352:0:64143:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572299.543355:0:64143:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0986 from it ffffb68a41fcfcb8 00010000:00010000:2.0:1788572299.543359:0:64143:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0986 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df826 expref: -99 pid: 64143 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572299.543367:0:64143:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0986 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df826 expref: -99 pid: 64143 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788572299.546415:0:64143:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788572299.546426:0:64143:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:3.0:1788572299.546452:0:64143:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572299.546471:0:64143:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0986 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df826 expref: -99 pid: 64143 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572299.546487:0:64143:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0986 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df826 expref: -99 pid: 64143 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572299.546495:0:64143:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572299.546503:0:64143:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:3.0:1788572299.546507:0:64143:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0986 00000080:00010000:3.0:1788572299.546515:0:64143:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0986 from it ffffb68a41fcfcb8 00010000:00010000:3.0:1788572299.546519:0:64143:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0986 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df826 expref: -99 pid: 64143 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572299.546528:0:64143:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0986 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df826 expref: -99 pid: 64143 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572299.546693:0:64143:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:3.0:1788572299.546701:0:64143:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572299.546708:0:64143:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0986 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df826 expref: -99 pid: 64143 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572299.546716:0:64143:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0986 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df826 expref: -99 pid: 64143 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572299.546723:0:64143:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572299.546727:0:64143:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:3.0:1788572299.546730:0:64143:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0986 00000080:00010000:3.0:1788572299.546733:0:64143:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0986 from it ffffb68a41fcfbb0 00010000:00010000:3.0:1788572299.546737:0:64143:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0986 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df826 expref: -99 pid: 64143 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572299.546743:0:64143:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0986 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df826 expref: -99 pid: 64143 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788572299.546751:0:64143:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 44 00000080:00200008:3.0:1788572299.546757:0:64143:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 100000 00000080:00200080:3.0:1788572299.546856:0:64143:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffca7ae0fe0 00800000:00000080:3.0:1788572299.546870:0:64143:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f32000 uarg=0000000000000000 00800000:00000004:3.0:1788572299.546877:0:64143:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0000-mdc-ffff94d405cc3800: age 3224, max_age 3451 00000100:00100000:3.0:1788572299.546921:0:64143:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:64143:1875450488468864:192.168.202.125@tcp:41:lfs.0 00000100:00100000:3.0:1788572299.547139:0:64143:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69300 going to sleep for 11 seconds 00000080:00200000:3.0:1788572299.628545:0:63659:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:3.0:1788572299.628577:0:63659:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572299.628597:0:63659:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0986 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df826 expref: -99 pid: 64143 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572299.628614:0:63659:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0986 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df826 expref: -99 pid: 64143 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788572299.628621:0:63659:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572299.628629:0:63659:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:3.0:1788572299.628633:0:63659:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0986 00000080:00010000:3.0:1788572299.628641:0:63659:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0986 from it ffffb68a41067b58 00010000:00010000:3.0:1788572299.628645:0:63659:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0986 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df826 expref: -99 pid: 64143 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572299.628655:0:63659:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0986 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df826 expref: -99 pid: 64143 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1788572299.628673:0:63659:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:3.0:1788572299.628695:0:63659:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f121.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), flags=9 00000080:00200000:3.0:1788572299.628699:0:63659:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f121.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), intent=getattr 00000002:00010000:3.0:1788572299.628712:0:63659:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f121.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788572299.628754:0:63659:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc098d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 63659 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572299.628763:0:63659:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc098d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 63659 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572299.628772:0:63659:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc098d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 63659 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572299.628797:0:63659:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job multiop:9bf287c0-f582-4b83-84c1-15e766218620:63659:1875450488469376:192.168.202.125@tcp:101:multiop.0 00000100:00100000:3.0:1788572299.628881:0:63659:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572299.638062:0:63659:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job multiop:9bf287c0-f582-4b83-84c1-15e766218620:63659:1875450488469376:192.168.202.125@tcp:101:multiop.0 00010000:00010000:2.0:1788572299.638093:0:63659:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc098d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df842 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572299.638115:0:63659:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc098d lrc: 4/1,0 mode: --/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df842 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572299.638125:0:63659:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc098d lrc: 4/1,0 mode: --/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df842 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572299.638138:0:63659:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc098d lrc: 4/1,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df842 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572299.638149:0:63659:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc098d lrc: 4/1,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df842 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572299.638162:0:63659:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc098d lrc: 5/2,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df842 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572299.638170:0:63659:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc098d lrc: 5/2,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df842 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572299.638177:0:63659:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc098d lrc: 4/1,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df842 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572299.638189:0:63659:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408c56a00 x1875450488469376/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 592/640 e 0 to 0 dl 1788572315 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572299.638224:0:63659:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc098d lrc: 3/1,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df842 expref: -99 pid: 63659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572299.638237:0:63659:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc098d lrc: 3/1,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df842 expref: -99 pid: 63659 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572299.638248:0:63659:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589975617/1 (0 0) 00000080:00200000:2.0:1788572299.638254:0:63659:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572299.638452:0:63659:0:(namei.c:147:ll_iget()) got inode: [0x20000a041:0x1:0x0](ffff94d410bcaa08): rc = 0 00000080:00200000:2.0:1788572299.638474:0:63659:0:(file.c:6826:ll_layout_conf()) [0x20000a041:0x1:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1788572299.638485:0:63659:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a041:0x1:0x0](ffff94d410bcaa08) for lock 0x1e260726fbbc098d 00000080:00010000:2.0:1788572299.638496:0:63659:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a041:0x1:0x0](ffff94d410bcaa08) 00000080:00010000:2.0:1788572299.638499:0:63659:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc098d from it ffffb68a41067b20 00010000:00010000:2.0:1788572299.638502:0:63659:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc098d lrc: 3/1,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4df842 expref: -99 pid: 63659 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572299.638511:0:63659:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc098d lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4df842 expref: -99 pid: 63659 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788572299.638545:0:63659:0:(file.c:6285:ll_getattr_dentry()) START file f121.replay-single:[0x20000a041:0x1:0x0](ffff94d410bcaa08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572299.638552:0:63659:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff94d410bcaa08),name=f121.replay-single 00000002:00010000:2.0:1788572299.638571:0:63659:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x20000a041:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572299.638579:0:63659:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc098d lrc: 2/1,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4df842 expref: -99 pid: 63659 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572299.638589:0:63659:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc098d lrc: 2/1,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4df842 expref: -99 pid: 63659 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572299.638596:0:63659:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572299.638600:0:63659:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a041:0x1:0x0](ffff94d410bcaa08) 00000080:00010000:2.0:1788572299.638603:0:63659:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a041:0x1:0x0](ffff94d410bcaa08) for lock 0x1e260726fbbc098d 00000080:00010000:2.0:1788572299.638606:0:63659:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc098d from it ffffb68a41067cb8 00010000:00010000:2.0:1788572299.638609:0:63659:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc098d lrc: 3/1,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4df842 expref: -99 pid: 63659 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572299.638616:0:63659:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc098d lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4df842 expref: -99 pid: 63659 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572299.638646:0:63659:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a041:0x1:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788572299.638654:0:63659:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000a041:0x1:0x0] 00000008:00010000:2.0:1788572299.638678:0:63659:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572299.638681:0:63659:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff94d4057bd730, osc lock ffff94d40857a390, flags 0x400001002 00010000:00010000:2.0:1788572299.638702:0:63659:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000047144dbc/0x1e260726fbbc0963 lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x682:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4df405 expref: -99 pid: 63659 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572299.638713:0:63659:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000047144dbc/0x1e260726fbbc0963 lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x682:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4df405 expref: -99 pid: 63659 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1788572299.638725:0:63659:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788572299.638729:0:63659:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000047144dbc/0x1e260726fbbc0963 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x682:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4df405 expref: -99 pid: 63659 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572299.638737:0:63659:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000047144dbc/0x1e260726fbbc0963 lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x682:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4df405 expref: -99 pid: 63659 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00200000:2.0:1788572299.638747:0:63659:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788572299.638755:0:63659:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000a041:0x1:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1788572299.638758:0:63659:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff94d4120272e0 (ffff94d405cee368/0) at cl_lock_release():245 00000020:00010000:2.0:1788572299.638762:0:63659:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d4120272e0 (ffff94d405cee368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572299.638765:0:63659:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d4057bd730 (ffff94d405ceeea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572299.638769:0:63659:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d4120272e0 (ffff94d405cee368/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572299.638771:0:63659:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d4057bd730 (ffff94d405cee368/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572299.638790:0:63659:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a041:0x1:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:2.0:1788572299.918782:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572300.933694:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572301.962426:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572302.981767:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:1.0:1788572303.365122:0:64261:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572303.365153:0:64261:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572303.365173:0:64261:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0986 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df826 expref: -99 pid: 64143 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572303.365190:0:64261:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0986 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df826 expref: -99 pid: 64143 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788572303.365198:0:64261:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572303.365232:0:64261:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572303.365237:0:64261:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0986 00000080:00010000:1.0:1788572303.365245:0:64261:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0986 from it ffffb68a42027b58 00010000:00010000:1.0:1788572303.365249:0:64261:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0986 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df826 expref: -99 pid: 64143 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572303.365258:0:64261:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0986 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df826 expref: -99 pid: 64143 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1788572303.365274:0:64261:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572303.365282:0:64261:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f121.replay-single, flags=8 00000080:00200008:1.0:1788572303.365291:0:64261:0:(file.c:6285:ll_getattr_dentry()) START file f121.replay-single:[0x20000a041:0x1:0x0](ffff94d410bcaa08), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572303.365296:0:64261:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x20000a041:0x1:0x0](ffff94d410bcaa08),name=f121.replay-single 00000002:00010000:1.0:1788572303.365308:0:64261:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x20000a041:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572303.365316:0:64261:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc098d lrc: 2/1,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4df842 expref: -99 pid: 63659 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572303.365325:0:64261:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc098d lrc: 2/1,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4df842 expref: -99 pid: 63659 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572303.365331:0:64261:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572303.365336:0:64261:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a041:0x1:0x0](ffff94d410bcaa08) 00000080:00010000:1.0:1788572303.365338:0:64261:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a041:0x1:0x0](ffff94d410bcaa08) for lock 0x1e260726fbbc098d 00000080:00010000:1.0:1788572303.365342:0:64261:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc098d from it ffffb68a42027cb8 00010000:00010000:1.0:1788572303.365345:0:64261:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc098d lrc: 3/1,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4df842 expref: -99 pid: 63659 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572303.365354:0:64261:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc098d lrc: 2/0,0 mode: PR/PR res: [0x20000a041:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4df842 expref: -99 pid: 63659 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788572303.365404:0:64261:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a041:0x1:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:1.0:1788572303.365413:0:64261:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000a041:0x1:0x0] 00000008:00010000:1.0:1788572303.365440:0:64261:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1788572303.365443:0:64261:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff94d408c7e030, osc lock ffff94d4124bd1d0, flags 0x400001002 00010000:00010000:1.0:1788572303.365463:0:64261:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000047144dbc/0x1e260726fbbc0963 lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x682:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4df405 expref: -99 pid: 63659 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788572303.365475:0:64261:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000047144dbc/0x1e260726fbbc0963 lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x682:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4df405 expref: -99 pid: 63659 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00200000:1.0:1788572303.367000:0:64261:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:1.0:1788572303.367006:0:64261:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000047144dbc/0x1e260726fbbc0963 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x682:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4df405 expref: -99 pid: 63659 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788572303.367018:0:64261:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000047144dbc/0x1e260726fbbc0963 lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x682:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4df405 expref: -99 pid: 63659 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000020:00200000:1.0:1788572303.367037:0:64261:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:2.0:1788572304.009358:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572305.029585:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572307.077387:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572307.658383:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488470400:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572307.658481:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488470656:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572307.658508:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488470912:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572307.658534:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788572307.658539:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488470528:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572307.658560:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488470784:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572307.672058:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488470528:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572307.672082:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488470784:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572307.672093:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488470400:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572307.672104:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488470656:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572307.672117:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488470912:192.168.202.125@tcp:400:kworker.0 00010000:00010000:3.0:1788572308.102998:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572309.019316:0:2396:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.125@tcp (source: 12345-192.168.202.125@tcp) 00000100:00100000:2.0:1788572309.019437:0:42440:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875450511393792 00000100:00100000:2.0:1788572309.019448:0:42440:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875450511393792 00000100:00100000:2.0:1788572309.019509:0:42440:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.125@tcp, seq: 88 00000100:00100000:2.0:1788572309.019520:0:42440:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff94d408c54e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8434:x1875450511393792:12345-192.168.202.125@tcp:104: 00010000:00010000:2.0:1788572309.019539:0:42440:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000047144dbc/0x1e260726fbbc0963 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x682:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0xec901abfeb4df405 expref: -99 pid: 63659 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572309.019617:0:42440:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788572309.019623:0:42440:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff94d408c54e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8434:x1875450511393792:12345-192.168.202.125@tcp:104: Request processed in 106us (318us total) trans 0 rc 0/0 00000100:00100000:2.0:1788572309.019630:0:42440:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.125@tcp, seq: 88 00000100:00100000:1.0:1788572309.020803:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488471040:192.168.202.125@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788572309.131395:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572310.155999:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572312.199403:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572312.773828:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488471168:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572312.773932:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488471424:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572312.773960:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488471680:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572312.773987:0:2405:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:2.0:1788572312.773995:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488471296:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572312.774016:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488471552:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572312.782372:0:2405:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 94489280513 gen 4 00000100:00100000:3.0:1788572312.782417:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488471296:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572312.782434:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488471424:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572312.785550:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488471552:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572312.785566:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488471680:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572312.790831:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488471168:192.168.202.125@tcp:400:kworker.0 00010000:00010000:3.0:1788572313.222390:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:1.0:1788572313.367575:0:64851:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572313.369290:0:64851:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572313.369322:0:64851:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572313.369460:0:64851:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0994 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 64851 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572313.369477:0:64851:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0994 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 64851 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572313.369487:0:64851:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0994 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 64851 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572313.369512:0:64851:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:64851:1875450488471808:192.168.202.125@tcp:101:lfs.0 00000100:00100000:1.0:1788572313.369605:0:64851:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d40504f0c0 going to sleep for 11 seconds 00000080:00200000:1.0:1788572313.493927:0:64861:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572313.493954:0:64861:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572313.494016:0:64861:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583c00/0x1e260726fbbc099b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 64861 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572313.494027:0:64861:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583c00/0x1e260726fbbc099b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 64861 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572313.494036:0:64861:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583c00/0x1e260726fbbc099b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 64861 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572313.494060:0:64861:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:64861:1875450488472832:192.168.202.125@tcp:101:rm.0 00000100:00100000:1.0:1788572313.494225:0:64861:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d40504fd80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572313.502972:0:64861:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:64861:1875450488472832:192.168.202.125@tcp:101:rm.0 00010000:00010000:1.0:1788572313.503001:0:64861:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583c00/0x1e260726fbbc099b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df88f expref: -99 pid: 64861 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572313.503015:0:64861:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583c00/0x1e260726fbbc099b lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df88f expref: -99 pid: 64861 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572313.503032:0:64861:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583c00/0x1e260726fbbc099b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df88f expref: -99 pid: 64861 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572313.503934:0:64861:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583c00/0x1e260726fbbc099b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df88f expref: -99 pid: 64861 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572313.503954:0:64861:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583c00/0x1e260726fbbc099b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df88f expref: -99 pid: 64861 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572313.503963:0:64861:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583c00/0x1e260726fbbc099b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df88f expref: -99 pid: 64861 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572313.503971:0:64861:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583c00/0x1e260726fbbc099b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4df88f expref: -99 pid: 64861 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572313.503984:0:64861:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d4049bf480 x1875450488472832/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572329 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572313.503999:0:64861:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583c00/0x1e260726fbbc099b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4df88f expref: -99 pid: 64861 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572313.504012:0:64861:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788572313.504021:0:64861:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00010000:00010000:3.0:1788572315.271070:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572316.293614:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572318.348553:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572319.367434:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572320.393521:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572321.413430:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788572321.496588:0:64927:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 00000100:00100000:2.0:1788572321.567295:0:2396:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.125@tcp (source: 12345-192.168.202.125@tcp) 00000100:00100000:3.0:1788572321.567620:0:42439:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875450511398912 00000100:00100000:3.0:1788572321.567633:0:42439:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875450511398912 00000100:00100000:3.0:1788572321.567696:0:42439:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.125@tcp, seq: 89 00000100:00100000:3.0:1788572321.567707:0:42439:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff94d408c61880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:9bf287c0-f582-4b83-84c1-15e766218620+5:80444:x1875450511398912:12345-192.168.202.125@tcp:104: 00010000:00010000:3.0:1788572321.567728:0:42439:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167e00/0x1e260726fbbc09b7 lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xec901abfeb4df906 expref: -99 pid: 64928 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572321.567815:0:42439:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788572321.567823:0:42439:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff94d408c61880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:9bf287c0-f582-4b83-84c1-15e766218620+5:80444:x1875450511398912:12345-192.168.202.125@tcp:104: Request processed in 118us (541us total) trans 0 rc 0/0 00000100:00100000:3.0:1788572321.567831:0:42439:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.125@tcp, seq: 89 00010000:00010000:1.0:1788572321.567907:0:63655:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788572321.567915:0:63655:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167e00/0x1e260726fbbc09b7 lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xec901abfeb4df906 expref: -99 pid: 64928 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572321.567924:0:63655:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412167e00 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:1.0:1788572321.567929:0:63655:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167e00/0x1e260726fbbc09b7 lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xec901abfeb4df906 expref: -99 pid: 64928 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572321.567937:0:63655:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167e00/0x1e260726fbbc09b7 lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xec901abfeb4df906 expref: -99 pid: 64928 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572321.567957:0:63655:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167e00/0x1e260726fbbc09b7 lrc: 3/0,0 mode: PR/PR res: [0x20000a811:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0xec901abfeb4df906 expref: -99 pid: 64928 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572321.568020:0:63655:0:(vvp_object.c:117:vvp_conf_set()) [0x20000a811:0x2:0x0]: losing layout lock 00010000:00010000:1.0:1788572321.568086:0:63655:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167e00/0x1e260726fbbc09b7 lrc: 2/0,0 mode: --/PR res: [0x20000a811:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xec901abfeb4df906 expref: -99 pid: 64928 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572321.568093:0:63655:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572321.568125:0:63655:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167e00/0x1e260726fbbc09b7 lrc: 1/0,0 mode: --/PR res: [0x20000a811:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xec901abfeb4df906 expref: -99 pid: 64928 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572321.568132:0:63655:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167e00/0x1e260726fbbc09b7 lrc: 0/0,0 mode: --/PR res: [0x20000a811:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xec901abfeb4df906 expref: -99 pid: 64928 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572321.568151:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572321.568153:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572321.568155:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572321.568163:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788572321.568178:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488474752:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572321.572016:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488474752:192.168.202.125@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1788572322.438822:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572323.461474:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572324.487777:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572325.509430:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572326.533405:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572327.557868:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572328.133625:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488476032:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572328.134447:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488476160:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572328.134495:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488476416:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572328.134875:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488476288:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572328.142942:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488476544:192.168.202.125@tcp:400:kworker.0 00000100:00080000:3.0:1788572328.154030:0:2405:0:(recover.c:205:ptlrpc_request_handle_notconn()) import lustre-MDT0000-mdc-ffff94d405cc3800 of lustre-MDT0000_UUID@192.168.202.125@tcp abruptly disconnected: reconnecting 00000100:00080000:3.0:1788572328.154038:0:2405:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:3.0:1788572328.154044:0:2405:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff94d405cc3800: Connection to lustre-MDT0000 (at 192.168.202.125@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:3.0:1788572328.154051:0:2405:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572328.154055:0:2405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3450 00000100:00080000:3.0:1788572328.154066:0:2405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572328.154077:0:2405:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572328.154106:0:2405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488476672/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572328.154138:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488476160:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572328.154195:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488476288:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572328.154316:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488476672:192.168.202.125@tcp:38:ptlrpcd_01_00.0 00000100:00100000:2.0:1788572328.155432:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488476544:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572328.155614:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488476416:192.168.202.125@tcp:400:kworker.0 00000100:00080000:1.0:1788572328.158313:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572328.158329:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-19) 00000100:00100000:1.0:1788572328.158334:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488476672:192.168.202.125@tcp:38:ptlrpcd_01_00.0 00010000:00010000:1.0:1788572328.584048:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572329.606954:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572330.629650:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572331.653516:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572332.677415:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572333.256286:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488476928:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788572333.256350:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488477184:192.168.202.125@tcp:9:kworker.0 00000100:00100000:3.0:1788572333.256374:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488477312:192.168.202.125@tcp:9:kworker.0 00000100:00080000:3.0:1788572333.259676:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572333.259689:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-19) 00000100:00100000:3.0:1788572333.259694:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488476928:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788572333.259715:0:2402:0:(import.c:2011:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff94d408d3e300 x1875450488477312/t0(0) o9->lustre-OST0001-osc-ffff94d405cc3800@192.168.202.125@tcp:28/4 lens 224/224 e 0 to 0 dl 1788572343 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572333.259725:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab800 lustre-OST0001_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:3.0:1788572333.259729:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488477312:192.168.202.125@tcp:9:kworker.0 00000100:00080000:3.0:1788572333.263446:0:2402:0:(import.c:2011:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff94d408d3ed80 x1875450488477184/t0(0) o9->lustre-OST0000-osc-ffff94d405cc3800@192.168.202.125@tcp:28/4 lens 224/224 e 0 to 0 dl 1788572343 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572333.263461:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bad000 lustre-OST0000_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:3.0:1788572333.263466:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488477184:192.168.202.125@tcp:9:kworker.0 00010000:00010000:1.0:1788572333.701419:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572334.726591:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572335.749596:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572336.779530:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572337.802274:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572338.374231:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488477440:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572338.374275:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488477568:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572338.374340:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788572338.374349:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488477696:192.168.202.125@tcp:400:kworker.0 00000100:00080000:3.0:1788572338.395285:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1788572338.395294:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488477696:192.168.202.125@tcp:400:kworker.0 00000100:00080000:3.0:1788572338.395368:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-19) 00000100:00100000:3.0:1788572338.395372:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488477568:192.168.202.125@tcp:38:kworker.0 00010000:00010000:3.0:1788572338.821523:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572339.845739:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572340.869599:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572341.899996:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572342.917522:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572343.493896:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572343.493903:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572343.493907:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572343.493918:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3491 00000100:00080000:3.0:1788572343.493930:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1788572343.493938:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572343.493949:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572343.493969:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488477952/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572343.494276:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488477824:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572343.494352:0:2405:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788572343.494358:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488478080:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572343.494638:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488477952:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788572343.499270:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488478080:192.168.202.125@tcp:400:kworker.0 00000100:00080000:1.0:1788572343.515690:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572343.515707:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-19) 00000100:00100000:1.0:1788572343.515714:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488477952:192.168.202.125@tcp:38:kworker.0 00010000:00010000:1.0:1788572343.943955:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1788572344.517495:0:2406:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572328/real 1788572328] req@ffff94d408c57480 x1875450488476032/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572344 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572344.517528:0:2406:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff94d408c57480 x1875450488476032/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572344 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572344.517536:0:2406:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from FULL to DISCONN 00000100:02020000:2.0:1788572344.517540:0:2406:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.202.125@tcp: Connection to MGS (at 192.168.202.125@tcp) was lost; in progress operations using this service will fail 00000100:00080000:2.0:1788572344.551431:0:2406:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.202.125@tcp for MGC192.168.202.125@tcp not replayable, auto-deactivating 00000100:00080000:2.0:1788572344.551438:0:2406:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:2.0:1788572344.551447:0:2406:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff94d408c57480 x1875450488476032/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572344 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572344.551478:0:2406:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff94d408d3f800 x1875450488476800/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572349 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572344.551491:0:2406:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff94d408d3dc00 x1875450488477440/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572354 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572344.551500:0:2406:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff94d408c57100 x1875450488477824/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572359 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:2.0:1788572344.551518:0:2406:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:2.0:1788572344.551521:0:2406:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:2.0:1788572344.551523:0:2406:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:2.0:1788572344.551525:0:2406:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788572344.551564:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) e60431de-58b9-4f69-9048-11a3d82ddcdd->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572344.551568:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572344.551571:0:470:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572344.551576:0:470:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3258 00000100:00080000:2.0:1788572344.551591:0:470:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d40490a800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788572344.551640:0:470:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488478208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572344.551661:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572344.551665:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572344.551667:0:470:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572344.551671:0:470:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3496 00000100:00080000:2.0:1788572344.551676:0:470:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572344.551680:0:470:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572344.551690:0:470:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572344.551702:0:470:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60700 x1875450488478336/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572344.551749:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488476032:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572344.551767:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788572344.551774:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488478464:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572344.562959:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488477824:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572344.576425:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488478464:192.168.202.125@tcp:400:kworker.0 00010000:00010000:3.0:1788572344.965871:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572345.989566:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572347.013889:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572348.037665:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572349.064100:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572349.638418:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) e60431de-58b9-4f69-9048-11a3d82ddcdd->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572349.638428:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) e60431de-58b9-4f69-9048-11a3d82ddcdd->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788572349.638431:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572349.638435:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572349.638439:0:470:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572349.638449:0:470:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3497 00000100:00080000:2.0:1788572349.638460:0:470:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572349.638467:0:470:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572349.638477:0:470:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572349.638510:0:470:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488478592/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572349.638650:0:2405:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788572349.638663:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488478720:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572349.638880:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488478592:192.168.202.125@tcp:38:kworker.0 00000100:00080000:1.0:1788572349.642488:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572349.642504:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-19) 00000100:00100000:1.0:1788572349.642510:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488478592:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788572349.645391:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488478720:192.168.202.125@tcp:400:kworker.0 00010000:00010000:1.0:1788572350.086795:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572351.110040:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572352.134656:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572353.158392:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572354.181558:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572354.757463:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572344/real 1788572344] req@ffff94d408c62d80 x1875450488478208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572354 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572354.757500:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488478208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572354 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572354.757519:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572354.757539:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572354.757543:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e60431de-58b9-4f69-9048-11a3d82ddcdd:2402:1875450488478208:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788572354.757670:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) e60431de-58b9-4f69-9048-11a3d82ddcdd->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572354.757676:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572354.757679:0:470:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572354.757684:0:470:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3497 00000100:00080000:2.0:1788572354.757695:0:470:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 5s 00000100:00080000:2.0:1788572354.757702:0:470:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d40490a800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788572354.757737:0:470:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488478848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572354.757761:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572354.757765:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572354.757767:0:470:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572354.757772:0:470:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3503 00000100:00080000:2.0:1788572354.757775:0:470:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572354.757779:0:470:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572354.757789:0:470:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572354.757803:0:470:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488478976/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572354.757924:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e60431de-58b9-4f69-9048-11a3d82ddcdd:2402:1875450488478848:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788572354.758026:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488478976:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788572354.758593:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488479104:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572354.763638:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488479104:192.168.202.125@tcp:400:kworker.0 00000100:00080000:1.0:1788572354.763808:0:2402:0:(import.c:1132:ptlrpc_connect_interpret()) MGC192.168.202.125@tcp: connect to target with instance 0 10000000:01000000:1.0:1788572354.763822:0:2402:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:02000400:1.0:1788572354.763828:0:2402:0:(import.c:1284:ptlrpc_connect_interpret()) Evicted from MGS (at 192.168.202.125@tcp) after server handle changed from 0xec901abfeb4df540 to 0xec901abfeb4df9ae 00000100:00080000:1.0:1788572354.781802:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from CONNECTING to EVICTED 00000100:00080000:1.0:1788572354.781821:0:2402:0:(import.c:1674:ptlrpc_import_recovery_state_machine()) evicted from MGS@192.168.202.125@tcp; invalidating 00000100:00080000:1.0:1788572354.789157:0:65151:0:(import.c:1598:ptlrpc_invalidate_import_thread()) thread invalidate import MGC192.168.202.125@tcp to MGS@192.168.202.125@tcp 00000100:00100000:1.0:1788572354.789162:0:65151:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:1.0:1788572354.789165:0:65151:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00010000:00010000:1.0:1788572354.789182:0:65151:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.125@tcp lock: ffff94d412583e00/0x1e260726fbbc0971 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110e400000000 nid: local remote: 0xec901abfeb4df619 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572354.789194:0:65151:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.125@tcp lock: ffff94d412583e00/0x1e260726fbbc0971 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110f400000000 nid: local remote: 0xec901abfeb4df619 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572354.789226:0:65151:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:1.0:1788572354.789238:0:65151:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572354.789250:0:65151:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.125@tcp lock: ffff94d412583e00/0x1e260726fbbc0971 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x590f400000000 nid: local remote: 0xec901abfeb4df619 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572354.789267:0:65151:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.125@tcp lock: ffff94d412166200/0x1e260726fbbc097f lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0xec901abfeb4df627 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572354.789272:0:65151:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.125@tcp lock: ffff94d412166200/0x1e260726fbbc097f lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0xec901abfeb4df627 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572354.789278:0:65151:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:1.0:1788572354.789281:0:65151:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572354.789284:0:65151:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.125@tcp lock: ffff94d412166200/0x1e260726fbbc097f lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0xec901abfeb4df627 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572354.789296:0:65151:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.125@tcp lock: ffff94d400031800/0x1e260726fbbc096a lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0xec901abfeb4df612 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572354.789306:0:65151:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.125@tcp lock: ffff94d400031800/0x1e260726fbbc096a lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0xec901abfeb4df612 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572354.789312:0:65151:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788572354.789315:0:65151:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572354.789318:0:65151:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.125@tcp lock: ffff94d400031800/0x1e260726fbbc096a lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0xec901abfeb4df612 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572354.789332:0:65151:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.125@tcp lock: ffff94d400031600/0x1e260726fbbc0978 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0xec901abfeb4df620 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572354.789338:0:65151:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.125@tcp lock: ffff94d400031600/0x1e260726fbbc0978 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0xec901abfeb4df620 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572354.789344:0:65151:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 00010000:00010000:1.0:1788572354.789347:0:65151:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572354.789351:0:65151:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.125@tcp lock: ffff94d400031600/0x1e260726fbbc0978 lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0xec901abfeb4df620 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:1.0:1788572354.789364:0:65151:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from EVICTED to RECOVER 00000100:00080000:1.0:1788572354.789368:0:65151:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from RECOVER to FULL 10000000:01000000:1.0:1788572354.789371:0:65151:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:1.0:1788572354.789374:0:65151:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:02000000:1.0:1788572354.789378:0:65151:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) MGC192.168.202.125@tcp: Connection restored to 192.168.202.125@tcp (at 192.168.202.125@tcp) 00010000:00010000:2.0:1788572355.215065:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572356.229688:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572356.508285:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488479488:192.168.202.125@tcp:38:ll_cfg_requeue.0 00000100:00100000:3.0:1788572356.542071:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488479744:192.168.202.125@tcp:501:ll_cfg_requeue.0 00000100:00100000:3.0:1788572356.542141:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488479872:192.168.202.125@tcp:503:ll_cfg_requeue.0 00000100:00100000:3.0:1788572356.542244:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4082340c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572356.551844:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488479872:192.168.202.125@tcp:503:ll_cfg_requeue.0 00000040:00080000:3.0:1788572356.553095:0:65192:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 64768 count 51 00000020:01000000:2.0:1788572356.554380:0:42451:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 52-51 (rc=0) 10000000:01000000:2.0:1788572356.554410:0:42451:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.202.125@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:2.0:1788572356.554423:0:42451:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412167000/0x1e260726fbbc09cc lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4dfb36 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572356.554435:0:42451:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.125@tcp lock: ffff94d412167000/0x1e260726fbbc09cc lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4dfb36 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572356.554446:0:42451:0:(mgc_request.c:611:do_requeue()) updating log params 10000000:01000000:2.0:1788572356.554448:0:42451:0:(mgc_request.c:1801:mgc_process_log()) Process log params-ffff94d405cc3800 from 7 10000000:01000000:2.0:1788572356.554451:0:42451:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:2.0:1788572356.554503:0:42451:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412536400/0x1e260726fbbc09d3 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572356.554513:0:42451:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.125@tcp lock: ffff94d412536400/0x1e260726fbbc09d3 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572356.554524:0:42451:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.125@tcp lock: ffff94d412536400/0x1e260726fbbc09d3 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572356.554552:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488480000:192.168.202.125@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788572356.554692:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d41827f780 going to sleep for 11 seconds 00000100:00100000:3.0:1788572356.564471:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488480000:192.168.202.125@tcp:101:ll_cfg_requeue.0 00010000:00010000:3.0:1788572356.564498:0:42451:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.125@tcp lock: ffff94d412536400/0x1e260726fbbc09d3 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4dfb3d expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572356.564523:0:42451:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.125@tcp lock: ffff94d412536400/0x1e260726fbbc09d3 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xec901abfeb4dfb3d expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572356.564604:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488480128:192.168.202.125@tcp:501:ll_cfg_requeue.0 00000100:00100000:3.0:1788572356.564665:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408234cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572356.573508:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488480128:192.168.202.125@tcp:501:ll_cfg_requeue.0 00000100:00100000:3.0:1788572356.573576:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488480256:192.168.202.125@tcp:503:ll_cfg_requeue.0 00000100:00100000:3.0:1788572356.573669:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408234cc0 going to sleep for 11 seconds 00000020:01000000:3.0:1788572356.580274:0:42451:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 7-6 (rc=0) 10000000:01000000:3.0:1788572356.580295:0:42451:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.202.125@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:3.0:1788572356.580309:0:42451:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412536400/0x1e260726fbbc09d3 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4dfb3d expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572356.580322:0:42451:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.125@tcp lock: ffff94d412536400/0x1e260726fbbc09d3 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4dfb3d expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572356.580333:0:42451:0:(mgc_request.c:611:do_requeue()) updating log lustre-sptlrpc 10000000:01000000:3.0:1788572356.580336:0:42451:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-sptlrpc-ffff94d4113f0000 from 1 10000000:01000000:3.0:1788572356.580339:0:42451:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:3.0:1788572356.580400:0:42451:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d400030600/0x1e260726fbbc09da lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572356.580408:0:42451:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.125@tcp lock: ffff94d400030600/0x1e260726fbbc09da lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572356.580419:0:42451:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.125@tcp lock: ffff94d400030600/0x1e260726fbbc09da lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572356.580449:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488480384:192.168.202.125@tcp:101:ll_cfg_requeue.0 00000100:00100000:3.0:1788572356.580683:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408234180 going to sleep for 11 seconds 00010000:00010000:3.0:1788572357.257237:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572357.829550:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488480640:192.168.202.125@tcp:17:kworker.0 00000100:00100000:2.0:1788572357.846662:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488480640:192.168.202.125@tcp:17:kworker.0 00010000:00010000:2.0:1788572358.280077:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572359.304293:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572359.877760:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488480768:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572359.877863:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788572359.877880:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488480896:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572359.882276:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488480896:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572359.882302:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488480768:192.168.202.125@tcp:400:kworker.0 00010000:00010000:3.0:1788572360.329755:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572360.386301:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488474624:192.168.202.125@tcp:101:lfs.0 00000100:00080000:1.0:1788572360.394756:0:2402:0:(client.c:3425:ptlrpc_replay_interpret()) @@@ got reply req@ffff94d408d3e680 x1875450488474624/t94489280517(94489280517) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 736/608 e 0 to 0 dl 1788572376 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00080000:1.0:1788572360.394781:0:2402:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff94d408d3c000 x1875450488475008/t94489280518(94489280518) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/456 e 0 to 0 dl 1788572337 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788572360.394789:0:2402:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:1.0:1788572360.394792:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 94489280514 gen 4 00000100:00100000:1.0:1788572360.394798:0:2402:0:(client.c:3156:ptlrpc_free_committed()) @@@ stopping search req@ffff94d408d3c700 x1875450488473216/t94489280515(94489280515) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572376 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572360.394806:0:2402:0:(recover.c:53:ptlrpc_replay_next()) import ffff94d405bab000 from lustre-MDT0000_UUID committed 94489280514 last 94489280517 00000100:00080000:1.0:1788572360.394813:0:2402:0:(client.c:3540:ptlrpc_replay_req()) @@@ REPLAY req@ffff94d408d3c000 x1875450488475008/t94489280518(94489280518) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/456 e 0 to 0 dl 1788572337 ref 2 fl New:RQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788572360.394825:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488474624:192.168.202.125@tcp:101:lfs.0 00000100:00100000:1.0:1788572360.394846:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488475008:192.168.202.125@tcp:35:lfs.0 00000100:00080000:1.0:1788572360.409983:0:2402:0:(client.c:3425:ptlrpc_replay_interpret()) @@@ got reply req@ffff94d408d3c000 x1875450488475008/t94489280518(94489280518) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/456 e 0 to 0 dl 1788572376 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788572360.409999:0:2402:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:1.0:1788572360.410002:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 94489280514 gen 4 00000100:00100000:1.0:1788572360.410009:0:2402:0:(client.c:3156:ptlrpc_free_committed()) @@@ stopping search req@ffff94d408d3c700 x1875450488473216/t94489280515(94489280515) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572376 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572360.410029:0:2402:0:(recover.c:53:ptlrpc_replay_next()) import ffff94d405bab000 from lustre-MDT0000_UUID committed 94489280514 last 94489280518 00000100:00080000:1.0:1788572360.410037:0:2402:0:(client.c:3540:ptlrpc_replay_req()) @@@ REPLAY req@ffff94d408d3f100 x1875450488475264/t94489280519(94489280519) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/456 e 0 to 0 dl 1788572337 ref 2 fl New:RQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788572360.410051:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488475008:192.168.202.125@tcp:35:lfs.0 00000100:00100000:1.0:1788572360.410075:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488475264:192.168.202.125@tcp:35:lfs.0 00000100:00080000:1.0:1788572360.431126:0:2402:0:(client.c:3425:ptlrpc_replay_interpret()) @@@ got reply req@ffff94d408d3f100 x1875450488475264/t94489280519(94489280519) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/456 e 0 to 0 dl 1788572376 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788572360.431144:0:2402:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:1.0:1788572360.431148:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 94489280514 gen 4 00000100:00100000:1.0:1788572360.431155:0:2402:0:(client.c:3156:ptlrpc_free_committed()) @@@ stopping search req@ffff94d408d3c700 x1875450488473216/t94489280515(94489280515) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572376 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572360.431164:0:2402:0:(recover.c:53:ptlrpc_replay_next()) import ffff94d405bab000 from lustre-MDT0000_UUID committed 94489280514 last 94489280519 00000100:00080000:1.0:1788572360.431170:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:3.0:1788572360.432333:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488475264:192.168.202.125@tcp:35:lfs.0 00010000:00080000:1.0:1788572360.432398:0:65200:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff94d405cc3800 to lustre-MDT0000_UUID@192.168.202.125@tcp 00010000:00010000:1.0:1788572360.432403:0:65200:0:(ldlm_request.c:2888:ldlm_cancel_unused_locks_for_replay()) Dropping as many unused locks as possible before replay for namespace lustre-MDT0000-mdc-ffff94d405cc3800 (0) 00010000:00010000:1.0:1788572360.432409:0:65200:0:(ldlm_request.c:2901:ldlm_cancel_unused_locks_for_replay()) Canceled 0 unused locks from namespace lustre-MDT0000-mdc-ffff94d405cc3800 00000100:00080000:1.0:1788572360.436274:0:65200:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to REPLAY_WAIT 00000100:00100000:3.0:1788572360.436404:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488481024:192.168.202.125@tcp:400:ldlm_lock_repla.0 00000100:00100000:3.0:1788572360.458085:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 94489280519 gen 4 00000100:00080000:3.0:1788572360.458164:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from REPLAY_WAIT to RECOVER 00000100:00080000:3.0:1788572360.458177:0:2402:0:(client.c:3234:ptlrpc_resend_req()) @@@ going to resend req@ffff94d4049bc380 x1875450488481024/t0(0) o400->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572376 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572360.458311:0:2402:0:(client.c:3243:ptlrpc_resend_req()) @@@ it has reply, so skip it req@ffff94d4049bc380 x1875450488481024/t0(0) o400->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572376 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572360.458320:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from RECOVER to FULL 00000100:00100000:1.0:1788572360.904638:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488481152:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572360.904749:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488481408:192.168.202.125@tcp:400:kworker.0 00010000:00010000:2.0:1788572361.351012:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572362.375030:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572363.404092:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572364.421440:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572365.445442:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572366.038765:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488481536:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572366.038877:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488481792:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572366.038909:0:2405:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788572366.038916:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488481664:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572366.043244:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488481664:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572366.043263:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488481536:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572366.043275:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488481792:192.168.202.125@tcp:400:kworker.0 00000080:00200008:2.0:1788572366.184096:0:65253:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572366.184112:0:65253:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572366.184383:0:65253:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572366.184446:0:65253:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 65253 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572366.184457:0:65253:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 65253 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572366.184468:0:65253:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 65253 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572366.184498:0:65253:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:65253:1875450488481920:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572366.184585:0:65253:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d41827f9c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572366.187843:0:65253:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:65253:1875450488481920:192.168.202.125@tcp:101:lfs.0 00010000:00010000:1.0:1788572366.187874:0:65253:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572366.187887:0:65253:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572366.187897:0:65253:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572366.187907:0:65253:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572366.187920:0:65253:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572366.187928:0:65253:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572366.187936:0:65253:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572366.187948:0:65253:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408c61180 x1875450488481920/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572382 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572366.187963:0:65253:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572366.187974:0:65253:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788572366.187981:0:65253:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572366.188012:0:65253:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1788572366.188019:0:65253:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc09e1 00000080:00010000:1.0:1788572366.188024:0:65253:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572366.188028:0:65253:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc09e1 from it ffffb68a40d17cb8 00010000:00010000:1.0:1788572366.188031:0:65253:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572366.188040:0:65253:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:1.0:1788572366.188807:0:65253:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1788572366.188815:0:65253:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572366.188834:0:65253:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572366.188847:0:65253:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572366.188860:0:65253:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572366.188867:0:65253:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572366.188874:0:65253:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572366.188877:0:65253:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc09e1 00000080:00010000:1.0:1788572366.188882:0:65253:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc09e1 from it ffffb68a40d17cb8 00010000:00010000:1.0:1788572366.188885:0:65253:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572366.188892:0:65253:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572366.189007:0:65253:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572366.189016:0:65253:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572366.189022:0:65253:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572366.189031:0:65253:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572366.189038:0:65253:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572366.189042:0:65253:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572366.189044:0:65253:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc09e1 00000080:00010000:1.0:1788572366.189048:0:65253:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc09e1 from it ffffb68a40d17bb0 00010000:00010000:1.0:1788572366.189051:0:65253:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572366.189058:0:65253:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788572366.189066:0:65253:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 44 00000080:00200008:1.0:1788572366.189072:0:65253:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 100000 00000080:00200080:1.0:1788572366.189158:0:65253:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc76887bc0 00800000:00000080:1.0:1788572366.189172:0:65253:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57000 uarg=0000000000000000 00800000:00000004:1.0:1788572366.189179:0:65253:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0000-mdc-ffff94d405cc3800: age 3466, max_age 3518 00000100:00100000:1.0:1788572366.189241:0:65253:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:65253:1875450488482048:192.168.202.125@tcp:41:lfs.0 00000100:00100000:1.0:1788572366.189323:0:65253:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d40504f840 going to sleep for 11 seconds 00000100:00100000:1.0:1788572366.196848:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488482304:192.168.202.125@tcp:8:lfs.0 00000100:00100000:1.0:1788572366.207256:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-OST0000-osc-ffff94d405cc3800: committing for last_committed 4294968859 gen 14 00000100:00080000:1.0:1788572366.207276:0:2402:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff94d405cc3800: connect to target with instance 1 00000100:00080000:1.0:1788572366.207286:0:2402:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff94d405cc3800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:1.0:1788572366.207303:0:2402:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:1.0:1788572366.207308:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bad000 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00000080:00000004:1.0:1788572366.207320:0:2402:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:1.0:1788572366.207325:0:2402:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:1.0:1788572366.207342:0:2402:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000b66b0cb2): req@ffff94d408d3d880 x1875450488482432/t0(0) o13->lustre-OST0000-osc-ffff94d405cc3800@192.168.202.125@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572366.207369:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488482304:192.168.202.125@tcp:8:lfs.0 00000100:00100000:1.0:1788572366.216017:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488482560:192.168.202.125@tcp:8:lfs.0 00000100:00100000:1.0:1788572366.219449:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-OST0001-osc-ffff94d405cc3800: committing for last_committed 0 gen 15 00000100:00080000:1.0:1788572366.219466:0:2402:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff94d405cc3800: connect to target with instance 1 00000100:00080000:1.0:1788572366.219476:0:2402:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff94d405cc3800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:1.0:1788572366.219488:0:2402:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:1.0:1788572366.219492:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab800 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00000080:00000004:1.0:1788572366.219501:0:2402:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:1.0:1788572366.219506:0:2402:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:1.0:1788572366.219521:0:2402:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000b66b0cb2): req@ffff94d408d3d880 x1875450488482688/t0(0) o13->lustre-OST0001-osc-ffff94d405cc3800@192.168.202.125@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572366.219537:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488482560:192.168.202.125@tcp:8:lfs.0 00010000:00010000:3.0:1788572366.473351:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572367.495918:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572368.517500:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572369.541453:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572370.582041:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572371.151423:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488482816:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572371.151446:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488483072:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572371.151459:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488482944:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572371.151471:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488483200:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572371.156498:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488483328:192.168.202.125@tcp:400:kworker.0 00010000:00010000:2.0:1788572371.605010:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572372.613494:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788572372.880148:0:65535:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 00010000:00010000:2.0:1788572373.640412:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788572374.639094:0:65683:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 00010000:00010000:2.0:1788572374.661475:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572375.687048:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572376.273695:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488483456:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572376.273792:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488483712:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572376.273818:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488483968:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572376.273843:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:1.0:1788572376.273848:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488483584:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572376.273870:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488483840:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572376.279009:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488483840:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572376.279027:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488483456:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572376.279038:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488483968:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572376.282516:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488483584:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572376.282596:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488483712:192.168.202.125@tcp:400:kworker.0 00010000:00010000:2.0:1788572376.716532:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:1.0:1788572377.015886:0:65721:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572377.015903:0:65721:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572377.015933:0:65721:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572377.015956:0:65721:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572377.015973:0:65721:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788572377.015981:0:65721:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572377.015990:0:65721:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572377.015994:0:65721:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc09e1 00000080:00010000:1.0:1788572377.016002:0:65721:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc09e1 from it ffffb68a40d7fcb8 00010000:00010000:1.0:1788572377.016006:0:65721:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572377.016016:0:65721:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1788572377.143910:0:65738:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:3.0:1788572377.145051:0:65738:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572377.145074:0:65738:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1788572377.145089:0:65738:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1788572377.145097:0:65738:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572377.145105:0:65738:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:3.0:1788572377.145109:0:65738:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc09e1 00000080:00010000:3.0:1788572377.145117:0:65738:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc09e1 from it ffffb68a4147bbf8 00010000:00010000:3.0:1788572377.145122:0:65738:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1788572377.145130:0:65738:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:3.0:1788572377.145147:0:65738:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:3.0:1788572377.146757:0:65738:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f130a.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), file ffff94d4122bbb00, open_flags 18800, mode 0 opened 0 00000080:00200000:3.0:1788572377.146773:0:65738:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f130a.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), intent=open 00000002:00010000:3.0:1788572377.146804:0:65738:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f130a.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0304001 00010000:00010000:3.0:1788572377.146870:0:65738:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030800/0x1e260726fbbc09e8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 65738 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572377.146881:0:65738:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030800/0x1e260726fbbc09e8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 65738 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788572377.146895:0:65738:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788572377.146909:0:65738:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c56a00 x1875450488484736/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 752/131944 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788572377.146919:0:65738:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030800/0x1e260726fbbc09e8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 65738 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572377.148723:0:65738:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:65738:1875450488484736:192.168.202.125@tcp:101:lfs.0 00010000:00010000:1.0:1788572378.757450:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572379.784699:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572381.384721:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488484992:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572381.394524:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488485248:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572381.406267:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488485504:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572381.406390:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488484992:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572381.448863:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488485248:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572381.448910:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488485504:192.168.202.125@tcp:400:kworker.0 00010000:00010000:3.0:1788572381.832528:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572382.854805:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572383.879169:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572384.910960:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572385.925643:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1788572386.092220:0:66327:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572386.092252:0:66327:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572386.092269:0:66327:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:2.0:1788572386.092284:0:66327:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:2.0:1788572386.092292:0:66327:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572386.092301:0:66327:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572386.092305:0:66327:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc09e1 00000080:00010000:2.0:1788572386.092313:0:66327:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc09e1 from it ffffb68a41fe7b58 00010000:00010000:2.0:1788572386.092317:0:66327:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:2.0:1788572386.092324:0:66327:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:2.0:1788572386.092338:0:66327:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572386.092351:0:66327:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d130b.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), flags=13 00000080:00200000:2.0:1788572386.092356:0:66327:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d130b.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), intent=getattr 00000002:00010000:2.0:1788572386.092369:0:66327:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d130b.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572386.092879:0:66327:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537600/0x1e260726fbbc09ef lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 66327 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.092889:0:66327:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537600/0x1e260726fbbc09ef lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 66327 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.092901:0:66327:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537600/0x1e260726fbbc09ef lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 66327 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572386.092929:0:66327:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:66327:1875450488485632:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572386.093148:0:66327:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572386.100023:0:66327:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:66327:1875450488485632:192.168.202.125@tcp:101:lfs.0 00010000:00010000:2.0:1788572386.100054:0:66327:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537600/0x1e260726fbbc09ef lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 66327 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.100069:0:66327:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537600/0x1e260726fbbc09ef lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 66327 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.100075:0:66327:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537600/0x1e260726fbbc09ef lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 66327 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.100083:0:66327:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537600/0x1e260726fbbc09ef lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 66327 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.100091:0:66327:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537600/0x1e260726fbbc09ef lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 66327 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.100096:0:66327:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412537600 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:2.0:1788572386.100101:0:66327:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537600/0x1e260726fbbc09ef lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 66327 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.100118:0:66327:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537600/0x1e260726fbbc09ef lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 66327 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.100125:0:66327:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537600/0x1e260726fbbc09ef lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 66327 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.100132:0:66327:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572386.100140:0:66327:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537600/0x1e260726fbbc09ef lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 66327 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.100146:0:66327:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537600/0x1e260726fbbc09ef lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 66327 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572386.100164:0:66327:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff94d408c62a00 x1875450488485632/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 592/584 e 0 to 0 dl 1788572402 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572386.100178:0:66327:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1788572386.100238:0:66327:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788572386.100252:0:66327:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00010000:2.0:1788572386.100262:0:66327:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc09e1 from it ffffb68a41fe7930 00010000:00010000:2.0:1788572386.100266:0:66327:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788572386.100273:0:66327:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:2.0:1788572386.100347:0:66327:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572386.100361:0:66327:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572386.100368:0:66327:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:2.0:1788572386.100377:0:66327:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:2.0:1788572386.100384:0:66327:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572386.100389:0:66327:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572386.100407:0:66327:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc09e1 00000080:00010000:2.0:1788572386.100412:0:66327:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc09e1 from it ffffb68a41fe7be8 00010000:00010000:2.0:1788572386.100415:0:66327:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:2.0:1788572386.100422:0:66327:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:2.0:1788572386.100431:0:66327:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572386.100438:0:66327:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d130b.replay-single, flags=16392 00000080:00200008:2.0:1788572386.100462:0:66327:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788572386.100468:0:66327:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572386.100474:0:66327:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572386.100481:0:66327:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:2.0:1788572386.100490:0:66327:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:2.0:1788572386.100496:0:66327:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572386.100499:0:66327:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572386.100502:0:66327:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc09e1 00000080:00010000:2.0:1788572386.100505:0:66327:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc09e1 from it ffffb68a41fe7cb8 00010000:00010000:2.0:1788572386.100507:0:66327:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:2.0:1788572386.100515:0:66327:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200008:2.0:1788572386.101054:0:66327:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788572386.101059:0:66327:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572386.101068:0:66327:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572386.101075:0:66327:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:2.0:1788572386.101085:0:66327:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:2.0:1788572386.101092:0:66327:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572386.101097:0:66327:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572386.101100:0:66327:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc09e1 00000080:00010000:2.0:1788572386.101104:0:66327:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc09e1 from it ffffb68a41fe7cb8 00010000:00010000:2.0:1788572386.101107:0:66327:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:2.0:1788572386.101114:0:66327:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:2.0:1788572386.101253:0:66327:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572386.101263:0:66327:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572386.101270:0:66327:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:2.0:1788572386.101278:0:66327:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:2.0:1788572386.101284:0:66327:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572386.101288:0:66327:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572386.101290:0:66327:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc09e1 00000080:00010000:2.0:1788572386.101294:0:66327:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc09e1 from it ffffb68a41fe7bb0 00010000:00010000:2.0:1788572386.101297:0:66327:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:2.0:1788572386.101304:0:66327:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:2.0:1788572386.101311:0:66327:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 44 00000080:00200008:2.0:1788572386.101318:0:66327:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 300000 00000080:00200080:2.0:1788572386.101335:0:66327:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c03066f0 arg=7fffcec3cbb0 00000080:00200000:2.0:1788572386.101348:0:66327:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) name=d130b.replay-single stripe_offset=0 stripe_count=1, hash_type=0 00010000:00010000:2.0:1788572386.101372:0:66327:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:2.0:1788572386.101422:0:66327:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:2.0:1788572386.101430:0:66327:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572386.101432:0:66327:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09e1 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dfd7b expref: -99 pid: 65253 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000020:00100000:2.0:1788572386.101455:0:66327:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572386.101465:0:66327:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c62a00 x1875450488485888/t0(0) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 568/664 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572386.101486:0:66327:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:66327:1875450488485888:192.168.202.125@tcp:36:lfs.0 00000100:00100000:2.0:1788572386.101577:0:66327:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572386.122539:0:66327:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:66327:1875450488485888:192.168.202.125@tcp:36:lfs.0 00000080:00200000:2.0:1788572386.122584:0:66327:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x20000a811:0x3:0x0](ffff94d4034300c8) 00000080:00200000:2.0:1788572386.122590:0:66327:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x20000a811:0x3:0x0], updating i_size 4096 00000080:00200008:2.0:1788572386.122622:0:66327:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 300000 00000080:00200000:2.0:1788572386.140405:0:66328:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572386.140437:0:66328:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572386.140495:0:66328:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 66328 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.140511:0:66328:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 66328 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.140522:0:66328:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 66328 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572386.140563:0:66328:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:66328:1875450488486016:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572386.140641:0:66328:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572386.143977:0:66328:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:66328:1875450488486016:192.168.202.125@tcp:101:lfs.0 00010000:00010000:2.0:1788572386.144003:0:66328:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.144016:0:66328:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.144028:0:66328:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.144038:0:66328:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.144050:0:66328:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.144059:0:66328:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.144072:0:66328:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572386.144083:0:66328:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408c62d80 x1875450488486016/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572402 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572386.144096:0:66328:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.144117:0:66328:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788572386.144123:0:66328:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572386.144149:0:66328:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788572386.144153:0:66328:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc09f6 00000080:00010000:2.0:1788572386.144159:0:66328:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572386.144162:0:66328:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc09f6 from it ffffb68a41db3b58 00010000:00010000:2.0:1788572386.144165:0:66328:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.144174:0:66328:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572386.144195:0:66328:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572386.144237:0:66328:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d130b.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), flags=8 00000080:00200000:2.0:1788572386.144242:0:66328:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d130b.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), intent=getattr 00000002:00010000:2.0:1788572386.144263:0:66328:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d130b.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572386.144302:0:66328:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc09fd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 66328 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.144309:0:66328:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc09fd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 66328 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.144317:0:66328:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc09fd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 66328 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572386.144337:0:66328:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:66328:1875450488486144:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572386.144402:0:66328:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572386.147671:0:66328:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:66328:1875450488486144:192.168.202.125@tcp:101:lfs.0 00010000:00010000:2.0:1788572386.147698:0:66328:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc09fd lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe07 expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.147719:0:66328:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc09fd lrc: 4/1,0 mode: --/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe07 expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.147727:0:66328:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc09fd lrc: 4/1,0 mode: --/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe07 expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.147737:0:66328:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc09fd lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfe07 expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.147748:0:66328:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc09fd lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe07 expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.147761:0:66328:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc09fd lrc: 5/2,0 mode: PR/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfe07 expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.147774:0:66328:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc09fd lrc: 5/2,0 mode: PR/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfe07 expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.147781:0:66328:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc09fd lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfe07 expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572386.147792:0:66328:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408c62d80 x1875450488486144/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 592/688 e 0 to 0 dl 1788572402 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572386.147805:0:66328:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc09fd lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe07 expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.147815:0:66328:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589977617/3 (0 0) 00000080:00200000:2.0:1788572386.147821:0:66328:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572386.147849:0:66328:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x20000a811:0x3:0x0], updating i_size 4096 00000080:00200000:2.0:1788572386.147853:0:66328:0:(namei.c:147:ll_iget()) got inode: [0x20000a811:0x3:0x0](ffff94d4034300c8): rc = 0 00000080:00010000:2.0:1788572386.147857:0:66328:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a811:0x3:0x0](ffff94d4034300c8) for lock 0x1e260726fbbc09fd 00000080:00010000:2.0:1788572386.147869:0:66328:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a811:0x3:0x0](ffff94d4034300c8) 00000080:00010000:2.0:1788572386.147872:0:66328:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc09fd from it ffffb68a41db3b20 00010000:00010000:2.0:1788572386.147876:0:66328:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc09fd lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfe07 expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.147884:0:66328:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc09fd lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe07 expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788572386.147923:0:66328:0:(file.c:6285:ll_getattr_dentry()) START file d130b.replay-single:[0x20000a811:0x3:0x0](ffff94d4034300c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572386.147930:0:66328:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x20000a811:0x3:0x0](ffff94d4034300c8),name=d130b.replay-single 00000002:00010000:2.0:1788572386.147953:0:66328:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x20000a811:0x3:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572386.147961:0:66328:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc09fd lrc: 2/1,0 mode: PR/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe07 expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572386.147972:0:66328:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc09fd lrc: 2/1,0 mode: PR/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe07 expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572386.147978:0:66328:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572386.147983:0:66328:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a811:0x3:0x0](ffff94d4034300c8) 00000080:00010000:2.0:1788572386.147986:0:66328:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a811:0x3:0x0](ffff94d4034300c8) for lock 0x1e260726fbbc09fd 00000080:00010000:2.0:1788572386.147990:0:66328:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc09fd from it ffffb68a41db3cb8 00010000:00010000:2.0:1788572386.147993:0:66328:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc09fd lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfe07 expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572386.148000:0:66328:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc09fd lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe07 expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572386.148053:0:66328:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572386.148061:0:66328:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572386.148068:0:66328:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572386.148076:0:66328:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572386.148082:0:66328:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572386.148086:0:66328:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572386.148089:0:66328:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc09f6 00000080:00010000:2.0:1788572386.148092:0:66328:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc09f6 from it ffffb68a41db3bf8 00010000:00010000:2.0:1788572386.148095:0:66328:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572386.148103:0:66328:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572386.148124:0:66328:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572386.148131:0:66328:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d130b.replay-single, flags=267 00000080:00200000:2.0:1788572386.148137:0:66328:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x20000a811:0x3:0x0](ffff94d4034300c8), inode mode 41ed mask 44 00000080:00200008:2.0:1788572386.148145:0:66328:0:(file.c:1090:ll_file_open()) START file d130b.replay-single:[0x20000a811:0x3:0x0](ffff94d4034300c8), flags 300000 00000002:00010000:2.0:1788572386.148159:0:66328:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x20000a811:0x3:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0240300001 00010000:00010000:2.0:1788572386.148233:0:66328:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0a04 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 66328 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.148242:0:66328:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0a04 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 66328 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572386.148253:0:66328:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572386.148261:0:66328:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c62d80 x1875450488486400/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 736/131944 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572386.148270:0:66328:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0a04 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 66328 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572386.148290:0:66328:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:66328:1875450488486400:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572386.148457:0:66328:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572386.156694:0:66328:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:66328:1875450488486400:192.168.202.125@tcp:101:lfs.0 00010000:00010000:2.0:1788572386.156718:0:66328:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0a04 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 66328 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.156733:0:66328:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0a04 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 66328 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.156740:0:66328:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0a04 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 66328 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.156748:0:66328:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0a04 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 66328 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.156757:0:66328:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0a04 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 66328 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.156763:0:66328:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412536c00 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:2.0:1788572386.156768:0:66328:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0a04 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 66328 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.156776:0:66328:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0a04 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 66328 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.156783:0:66328:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0a04 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 66328 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.156790:0:66328:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572386.156796:0:66328:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0a04 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 66328 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572386.156803:0:66328:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0a04 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 66328 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572386.156821:0:66328:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff94d408c62d80 x1875450488486400/t98784247810(98784247810) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 736/664 e 0 to 0 dl 1788572402 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572386.156838:0:66328:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d408c62d80 x1875450488486400/t98784247810(98784247810) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 736/664 e 0 to 0 dl 1788572402 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572386.156862:0:66328:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572386.156889:0:66328:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x20000a811:0x3:0x0], updating i_size 4096 00000002:00100000:2.0:1788572386.156908:0:66328:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d408c62d80 x1875450488486400/t98784247810(98784247810) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572402 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572386.157011:0:66328:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x20000a811:0x3:0x0](ffff94d4034300c8), inode mode 41ed mask 2 00000080:00200000:2.0:1788572386.157024:0:66328:0:(xattr.c:413:ll_xattr_set()) VFS Op:inode=[0x20000a811:0x3:0x0](ffff94d4034300c8), xattr lov 00010000:00010000:2.0:1788572386.157046:0:66328:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc09fd lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dfe07 expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572386.157094:0:66328:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc09fd lrc: 1/0,0 mode: --/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dfe07 expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572386.157101:0:66328:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572386.157105:0:66328:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc09fd lrc: 0/0,0 mode: --/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dfe07 expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788572386.157133:0:66328:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572386.157140:0:66328:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c62300 x1875450488486656/t0(0) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 680/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572386.157160:0:66328:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:66328:1875450488486656:192.168.202.125@tcp:36:lfs.0 00000100:00100000:2.0:1788572386.157270:0:66328:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b696c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572386.163273:0:66328:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:66328:1875450488486656:192.168.202.125@tcp:36:lfs.0 00000080:00200008:3.0:1788572386.163338:0:66328:0:(file.c:479:ll_file_release()) START file d130b.replay-single:[0x20000a811:0x3:0x0](ffff94d4034300c8), flags 300000 00000002:00100000:3.0:1788572386.163401:0:66328:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff94d408c62d80 x1875450488486400/t98784247810(98784247810) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572402 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788572386.163414:0:66328:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788572386.163420:0:66328:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c54380 x1875450488486912/t0(0) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/712 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788572386.163442:0:66328:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:66328:1875450488486912:192.168.202.125@tcp:35:lfs.0 00000100:00100000:3.0:1788572386.163510:0:66328:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d412842480 going to sleep for 11 seconds 00000100:00100000:3.0:1788572386.166740:0:66328:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:66328:1875450488486912:192.168.202.125@tcp:35:lfs.0 00000002:00100000:3.0:1788572386.166779:0:66328:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff94d408c62d80 x1875450488486400/t98784247810(98784247810) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572402 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788572386.501812:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488487040:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572386.501924:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488487296:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572386.501965:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:2.0:1788572386.501972:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488487168:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572386.502010:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488487424:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572386.512421:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488487168:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572386.512441:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488487424:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572386.512451:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488487040:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572386.512461:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488487296:192.168.202.125@tcp:400:kworker.0 00010000:00010000:2.0:1788572386.949397:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:2.0:1788572387.150601:0:66338:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572387.150622:0:66338:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572387.150771:0:66338:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572387.150791:0:66338:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572387.150810:0:66338:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572387.150820:0:66338:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572387.150829:0:66338:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572387.150834:0:66338:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc09f6 00000080:00010000:2.0:1788572387.150841:0:66338:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc09f6 from it ffffb68a41de7cb8 00010000:00010000:2.0:1788572387.150846:0:66338:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572387.150856:0:66338:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200008:2.0:1788572387.151983:0:66338:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788572387.151993:0:66338:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572387.152011:0:66338:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572387.152026:0:66338:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572387.152039:0:66338:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572387.152047:0:66338:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572387.152054:0:66338:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572387.152058:0:66338:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc09f6 00000080:00010000:2.0:1788572387.152064:0:66338:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc09f6 from it ffffb68a41de7cb8 00010000:00010000:2.0:1788572387.152068:0:66338:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572387.152076:0:66338:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788572387.155042:0:66338:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572387.155075:0:66338:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572387.155094:0:66338:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572387.155109:0:66338:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788572387.155117:0:66338:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572387.155123:0:66338:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572387.155127:0:66338:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc09f6 00000080:00010000:2.0:1788572387.155148:0:66338:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc09f6 from it ffffb68a41de7bb0 00010000:00010000:2.0:1788572387.155153:0:66338:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572387.155161:0:66338:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1788572387.155177:0:66338:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 44 00000080:00200008:2.0:1788572387.155225:0:66338:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 100000 00000080:00200080:2.0:1788572387.155355:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.155371:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00800000:00000004:2.0:1788572387.155377:0:66338:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0000-mdc-ffff94d405cc3800: age 3530, max_age 3539 00000100:00100000:2.0:1788572387.155495:0:66338:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:66338:1875450488487552:192.168.202.125@tcp:41:lfs.0 00000100:00100000:2.0:1788572387.155901:0:66338:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572387.160891:0:66338:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 98784247812 gen 4 00000100:00100000:2.0:1788572387.160923:0:66338:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:66338:1875450488487552:192.168.202.125@tcp:41:lfs.0 00800000:00000004:2.0:1788572387.160944:0:66338:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0000-mdc-ffff94d405cc3800: update ffff94d412909e10 cache blocks 317131/353529 objects 1023605/1024000 00000080:00200080:2.0:1788572387.161022:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.161038:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00800000:00000004:2.0:1788572387.161045:0:66338:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0001-mdc-ffff94d405cc3800: age 3530, max_age 3539 00000100:00100000:2.0:1788572387.161075:0:66338:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:66338:1875450488487680:192.168.202.125@tcp:41:lfs.0 00000100:00100000:2.0:1788572387.161237:0:66338:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572387.178577:0:66338:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:66338:1875450488487680:192.168.202.125@tcp:41:lfs.0 00800000:00000004:2.0:1788572387.178617:0:66338:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0001-mdc-ffff94d405cc3800: update ffff94d407db9e10 cache blocks 317207/353529 objects 1023684/1024000 00000080:00200080:2.0:1788572387.178701:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178720:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178730:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178734:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178738:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178742:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178745:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178749:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178754:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178757:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178760:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178764:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178768:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178771:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178777:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178780:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178785:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178788:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178792:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178796:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178800:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178804:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178808:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178812:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178815:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178819:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178823:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178827:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178831:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178835:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178838:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178842:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178846:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178850:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178854:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178858:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178862:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178866:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178870:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178874:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178877:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178880:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178884:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178887:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178890:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178894:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178897:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178901:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178905:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178908:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178912:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178915:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178920:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178924:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178928:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178931:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178935:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178940:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178943:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178946:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178950:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178953:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178957:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178960:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178965:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178968:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178972:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178975:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178979:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178983:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178987:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178991:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.178994:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.178997:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.179001:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.179004:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.179008:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.179012:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.179016:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.179020:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.179024:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.179028:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.179032:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.179035:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.179039:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.179042:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.179047:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.179050:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.179054:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.179057:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.179061:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.179065:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.179068:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.179071:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.179075:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.179078:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.179083:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.179086:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.179089:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.179093:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.179097:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.179100:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.179105:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.179109:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.179113:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.179117:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.179121:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.179124:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.179128:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.179132:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.179136:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.179140:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.179144:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.179148:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.179152:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.179156:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.179160:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.179164:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.179168:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.179172:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.179175:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.179179:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.179183:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.179186:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.179189:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00800000:00000080:2.0:1788572387.179192:0:66338:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.179196:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.179247:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00020000:00000004:2.0:1788572387.179257:0:66338:0:(obd_class.h:1091:obd_statfs()) lustre-OST0000-osc-ffff94d405cc3800: age 3530, max_age 3539 00000100:00100000:2.0:1788572387.179298:0:66338:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:66338:1875450488487808:192.168.202.125@tcp:13:lfs.0 00000100:00100000:2.0:1788572387.179487:0:66338:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572387.183427:0:66338:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:66338:1875450488487808:192.168.202.125@tcp:13:lfs.0 00020000:00000004:2.0:1788572387.183460:0:66338:0:(obd_class.h:1117:obd_statfs()) lustre-OST0000-osc-ffff94d405cc3800: update ffff94d407dbd190 cache blocks 901369/958279 objects 261733/262144 00000080:00200080:2.0:1788572387.183538:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.183566:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00020000:00000004:2.0:1788572387.183574:0:66338:0:(obd_class.h:1091:obd_statfs()) lustre-OST0001-osc-ffff94d405cc3800: age 3530, max_age 3539 00000100:00100000:2.0:1788572387.183620:0:66338:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:66338:1875450488487936:192.168.202.125@tcp:13:lfs.0 00000100:00100000:2.0:1788572387.183823:0:66338:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572387.186978:0:66338:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:66338:1875450488487936:192.168.202.125@tcp:13:lfs.0 00020000:00000004:2.0:1788572387.187011:0:66338:0:(obd_class.h:1117:obd_statfs()) lustre-OST0001-osc-ffff94d405cc3800: update ffff94d407dbb7d0 cache blocks 901369/958279 objects 261733/262144 00000080:00200080:2.0:1788572387.187086:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187108:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187115:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187121:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187124:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187128:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187143:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187148:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187152:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187155:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187159:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187163:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187167:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187171:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187175:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187188:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187194:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187199:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187245:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187251:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187255:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187258:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187262:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187266:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187270:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187273:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187277:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187280:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187284:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187287:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187291:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187294:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187299:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187302:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187306:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187309:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187314:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187317:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187321:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187324:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187328:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187332:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187336:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187339:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187343:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187347:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187351:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187355:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187359:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187362:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187367:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187370:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187375:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187378:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187382:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187386:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187390:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187393:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187397:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187401:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187405:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187422:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187428:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187431:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187435:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187438:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187442:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187445:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187449:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187452:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187456:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187460:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187464:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187468:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187472:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187476:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187480:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187484:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187488:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187491:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187494:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187497:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187501:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187505:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187509:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187512:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187517:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187520:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187525:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187528:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187533:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187536:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187540:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187543:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187546:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187558:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187564:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187567:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187570:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187574:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187577:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187580:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187584:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187588:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187599:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187604:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187608:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187612:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187616:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187620:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187624:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187628:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200080:2.0:1788572387.187632:0:66338:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd50385d0 00020000:00000080:2.0:1788572387.187636:0:66338:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541f000 uarg=0000000000000000 00000080:00200000:3.0:1788572387.291662:0:66345:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:3.0:1788572387.291694:0:66345:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572387.291712:0:66345:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788572387.291728:0:66345:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788572387.291737:0:66345:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572387.291745:0:66345:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:3.0:1788572387.291749:0:66345:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc09f6 00000080:00010000:3.0:1788572387.291757:0:66345:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc09f6 from it ffffb68a41c37bd0 00010000:00010000:3.0:1788572387.291761:0:66345:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788572387.291770:0:66345:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1788572387.291786:0:66345:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:3.0:1788572387.291798:0:66345:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg225-client.virtnet, dir=[0x200000007:0x1:0x0](ffff94d403679988), flags=1536 00000080:00200000:3.0:1788572387.291801:0:66345:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:3.0:1788572387.291807:0:66345:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572387.291813:0:66345:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1788572387.291821:0:66345:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1788572387.291828:0:66345:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572387.291832:0:66345:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:3.0:1788572387.291835:0:66345:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc09f6 00000080:00010000:3.0:1788572387.291838:0:66345:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc09f6 from it ffffb68a41c37cd0 00010000:00010000:3.0:1788572387.291841:0:66345:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1788572387.291849:0:66345:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:3.0:1788572387.291861:0:66345:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 3 00000080:00200000:3.0:1788572387.291865:0:66345:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:3.0:1788572387.291870:0:66345:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572387.291875:0:66345:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1788572387.291883:0:66345:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:3.0:1788572387.291889:0:66345:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572387.291892:0:66345:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:3.0:1788572387.291895:0:66345:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc09f6 00000080:00010000:3.0:1788572387.291899:0:66345:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc09f6 from it ffffb68a41c37db0 00010000:00010000:3.0:1788572387.291901:0:66345:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1788572387.291908:0:66345:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:3.0:1788572387.291916:0:66345:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 3 00000080:00200000:3.0:1788572387.291921:0:66345:0:(namei.c:2129:ll_create_nd()) VFS Op:name=fsa-oleg225-client.virtnet, dir=[0x200000007:0x1:0x0](ffff94d403679988), flags=33188, excl=1 00000080:00200000:3.0:1788572387.291925:0:66345:0:(namei.c:2084:ll_mknod()) VFS Op:name=fsa-oleg225-client.virtnet, dir=[0x200000007:0x1:0x0](ffff94d403679988) mode 100644 dev 0 00010000:00010000:3.0:1788572387.291949:0:66345:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1788572387.291994:0:66345:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1788572387.292002:0:66345:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572387.292005:0:66345:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc09f6 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dfdeb expref: -99 pid: 66328 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000020:00100000:3.0:1788572387.292028:0:66345:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572387.292039:0:66345:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c54380 x1875450488488192/t0(0) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 528/664 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572387.292061:0:66345:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job mcreate:9bf287c0-f582-4b83-84c1-15e766218620:66345:1875450488488192:192.168.202.125@tcp:36:mcreate.0 00000100:00100000:3.0:1788572387.292146:0:66345:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4128423c0 going to sleep for 11 seconds 00000080:00200000:2.0:1788572387.322531:0:66348:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572387.322579:0:66348:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572387.322653:0:66348:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0a0b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 66348 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572387.322665:0:66348:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0a0b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 66348 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572387.322677:0:66348:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0a0b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 66348 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572387.322707:0:66348:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:66348:1875450488488320:192.168.202.125@tcp:101:rm.0 00000100:00100000:2.0:1788572387.322804:0:66348:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572387.327545:0:66348:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:66348:1875450488488320:192.168.202.125@tcp:101:rm.0 00010000:00010000:2.0:1788572387.327584:0:66348:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0a0b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe38 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572387.327597:0:66348:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0a0b lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe38 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572387.327620:0:66348:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0a0b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfe38 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572387.327632:0:66348:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0a0b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe38 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572387.327717:0:66348:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0a0b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfe38 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572387.327725:0:66348:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0a0b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfe38 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572387.327733:0:66348:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0a0b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfe38 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572387.327745:0:66348:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408c63800 x1875450488488320/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572403 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572387.327758:0:66348:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0a0b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe38 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572387.327771:0:66348:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788572387.327779:0:66348:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572387.327807:0:66348:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788572387.327814:0:66348:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0a0b 00000080:00010000:2.0:1788572387.327820:0:66348:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572387.327823:0:66348:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a0b from it ffffb68a41c37b58 00010000:00010000:2.0:1788572387.327826:0:66348:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0a0b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfe38 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572387.327834:0:66348:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0a0b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe38 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572387.327864:0:66348:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572387.327882:0:66348:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg225-client.virtnet, dir=[0x200000007:0x1:0x0](ffff94d403679988), flags=8 00000080:00200000:2.0:1788572387.327887:0:66348:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=fsa-oleg225-client.virtnet, dir=[0x200000007:0x1:0x0](ffff94d403679988), intent=getattr 00000002:00010000:2.0:1788572387.327907:0:66348:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: fsa-oleg225-client.virtnet,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572387.327947:0:66348:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0a12 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 66348 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572387.327956:0:66348:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0a12 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 66348 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572387.327966:0:66348:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0a12 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 66348 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572387.327988:0:66348:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:66348:1875450488488448:192.168.202.125@tcp:101:rm.0 00000100:00100000:2.0:1788572387.328074:0:66348:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572387.332892:0:66348:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:66348:1875450488488448:192.168.202.125@tcp:101:rm.0 00010000:00010000:2.0:1788572387.332921:0:66348:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0a12 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe54 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572387.332942:0:66348:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0a12 lrc: 4/1,0 mode: --/PR res: [0x20000a811:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe54 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572387.332950:0:66348:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0a12 lrc: 4/1,0 mode: --/PR res: [0x20000a811:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe54 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572387.332961:0:66348:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0a12 lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfe54 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572387.332972:0:66348:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0a12 lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe54 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572387.332985:0:66348:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0a12 lrc: 5/2,0 mode: PR/PR res: [0x20000a811:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfe54 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572387.332993:0:66348:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0a12 lrc: 5/2,0 mode: PR/PR res: [0x20000a811:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfe54 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572387.333000:0:66348:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0a12 lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfe54 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572387.333012:0:66348:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408c63800 x1875450488488448/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 600/584 e 0 to 0 dl 1788572403 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572387.333024:0:66348:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0a12 lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe54 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572387.333035:0:66348:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589977617/4 (0 0) 00000080:00200000:2.0:1788572387.333041:0:66348:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572387.333062:0:66348:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x20000a811:0x4:0x0], updating i_size 0 00000080:00200000:2.0:1788572387.333066:0:66348:0:(namei.c:147:ll_iget()) got inode: [0x20000a811:0x4:0x0](ffff94d410bc8908): rc = 0 00000080:00200000:2.0:1788572387.333107:0:66348:0:(file.c:6826:ll_layout_conf()) [0x20000a811:0x4:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1788572387.333131:0:66348:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a811:0x4:0x0](ffff94d410bc8908) for lock 0x1e260726fbbc0a12 00000080:00010000:2.0:1788572387.333142:0:66348:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a811:0x4:0x0](ffff94d410bc8908) 00000080:00010000:2.0:1788572387.333146:0:66348:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a12 from it ffffb68a41c37b20 00010000:00010000:2.0:1788572387.333149:0:66348:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0a12 lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4dfe54 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572387.333158:0:66348:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0a12 lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4dfe54 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788572387.333196:0:66348:0:(file.c:6285:ll_getattr_dentry()) START file fsa-oleg225-client.virtnet:[0x20000a811:0x4:0x0](ffff94d410bc8908), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572387.333220:0:66348:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x20000a811:0x4:0x0](ffff94d410bc8908),name=fsa-oleg225-client.virtnet 00000002:00010000:2.0:1788572387.333243:0:66348:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x20000a811:0x4:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572387.333253:0:66348:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0a12 lrc: 2/1,0 mode: PR/PR res: [0x20000a811:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4dfe54 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572387.333263:0:66348:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0a12 lrc: 2/1,0 mode: PR/PR res: [0x20000a811:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4dfe54 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572387.333269:0:66348:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572387.333274:0:66348:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a811:0x4:0x0](ffff94d410bc8908) 00000080:00010000:2.0:1788572387.333276:0:66348:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a811:0x4:0x0](ffff94d410bc8908) for lock 0x1e260726fbbc0a12 00000080:00010000:2.0:1788572387.333279:0:66348:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a12 from it ffffb68a41c37cb8 00010000:00010000:2.0:1788572387.333282:0:66348:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0a12 lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4dfe54 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572387.333288:0:66348:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0a12 lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4dfe54 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572387.333345:0:66348:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572387.333353:0:66348:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572387.333360:0:66348:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0a0b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe38 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572387.333373:0:66348:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0a0b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe38 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572387.333379:0:66348:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572387.333383:0:66348:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572387.333386:0:66348:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0a0b 00000080:00010000:2.0:1788572387.333389:0:66348:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a0b from it ffffb68a41c37c10 00010000:00010000:2.0:1788572387.333392:0:66348:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0a0b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfe38 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572387.333399:0:66348:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0a0b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe38 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572387.333407:0:66348:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572387.333428:0:66348:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=fsa-oleg225-client.virtnet, flags=0 00000080:00200000:2.0:1788572387.333433:0:66348:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572387.333440:0:66348:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572387.333446:0:66348:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0a0b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe38 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572387.333453:0:66348:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0a0b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe38 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572387.333460:0:66348:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572387.333464:0:66348:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572387.333466:0:66348:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0a0b 00000080:00010000:2.0:1788572387.333469:0:66348:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a0b from it ffffb68a41c37d68 00010000:00010000:2.0:1788572387.333472:0:66348:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0a0b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfe38 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572387.333479:0:66348:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0a0b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe38 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572387.333486:0:66348:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 3 00000080:00200000:2.0:1788572387.333491:0:66348:0:(namei.c:2418:ll_unlink()) VFS Op:name=fsa-oleg225-client.virtnet, dir=[0x200000007:0x1:0x0](ffff94d403679988) 00010000:00010000:2.0:1788572387.333528:0:66348:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0a0b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4dfe38 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572387.333551:0:66348:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0a12 lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xec901abfeb4dfe54 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572387.333579:0:66348:0:(vvp_object.c:117:vvp_conf_set()) [0x20000a811:0x4:0x0]: losing layout lock 00010000:00010000:2.0:1788572387.333627:0:66348:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0a12 lrc: 1/0,0 mode: --/PR res: [0x20000a811:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xec901abfeb4dfe54 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572387.333636:0:66348:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0a0b lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dfe38 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572387.333757:0:66348:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788572387.333761:0:66348:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0a12 lrc: 0/0,0 mode: --/PR res: [0x20000a811:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xec901abfeb4dfe54 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572387.333779:0:66348:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0a0b lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4dfe38 expref: -99 pid: 66348 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1788572387.333799:0:66348:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572387.333807:0:66348:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c63800 x1875450488488704/t0(0) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 512/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572387.333832:0:66348:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:66348:1875450488488704:192.168.202.125@tcp:36:rm.0 00000100:00100000:2.0:1788572387.333906:0:66348:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572387.340634:0:66348:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:66348:1875450488488704:192.168.202.125@tcp:36:rm.0 00000080:00200000:2.0:1788572387.340717:0:66348:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572387.340727:0:66348:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572387.340752:0:66348:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x20000a811:0x4:0x0](ffff94d410bc8908) 00010000:00010000:3.0:1788572387.975942:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572389.001463:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572390.022507:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572391.047143:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572391.622094:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488488960:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572391.622186:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488489216:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572391.655946:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488488960:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572391.655960:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488489216:192.168.202.125@tcp:400:kworker.0 00010000:00010000:1.0:1788572392.070083:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572393.093566:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572394.117722:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572395.141926:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788572395.361536:0:66414:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 00010000:00010000:0.0:1788572395.388383:0:66415:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0a19 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 66415 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572395.388399:0:66415:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0a19 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 66415 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572395.388411:0:66415:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0a19 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 66415 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572395.388440:0:66415:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job touch:9bf287c0-f582-4b83-84c1-15e766218620:66415:1875450488489472:192.168.202.125@tcp:101:touch.0 00000100:00100000:0.0:1788572395.388527:0:66415:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408234300 going to sleep for 11 seconds 00000100:00100000:0.0:1788572395.397322:0:66415:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job touch:9bf287c0-f582-4b83-84c1-15e766218620:66415:1875450488489472:192.168.202.125@tcp:101:touch.0 00010000:00010000:0.0:1788572395.397350:0:66415:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0a19 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe7e expref: -99 pid: 66415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572395.397363:0:66415:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0a19 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe7e expref: -99 pid: 66415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572395.397374:0:66415:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0a19 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfe7e expref: -99 pid: 66415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572395.397384:0:66415:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0a19 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe7e expref: -99 pid: 66415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572395.397397:0:66415:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0a19 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfe7e expref: -99 pid: 66415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572395.397404:0:66415:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0a19 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfe7e expref: -99 pid: 66415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572395.397414:0:66415:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0a19 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfe7e expref: -99 pid: 66415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572395.397424:0:66415:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408d3f100 x1875450488489472/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572411 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572395.397437:0:66415:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0a19 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe7e expref: -99 pid: 66415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572395.397448:0:66415:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788572395.397455:0:66415:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788572395.397484:0:66415:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1788572395.397489:0:66415:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0a19 00000080:00010000:0.0:1788572395.397493:0:66415:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572395.397496:0:66415:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a19 from it ffffb68a41f7fbf8 00010000:00010000:0.0:1788572395.397499:0:66415:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0a19 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfe7e expref: -99 pid: 66415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572395.397507:0:66415:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0a19 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe7e expref: -99 pid: 66415 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572395.397529:0:66415:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572395.397546:0:66415:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d130b.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), flags=25 00000080:00200000:0.0:1788572395.397551:0:66415:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d130b.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), intent=0 00000002:00010000:0.0:1788572395.397572:0:66415:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d130b.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1788572395.397607:0:66415:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167a00/0x1e260726fbbc0a20 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 66415 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572395.397614:0:66415:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167a00/0x1e260726fbbc0a20 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 66415 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572395.397623:0:66415:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167a00/0x1e260726fbbc0a20 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 66415 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572395.397641:0:66415:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job touch:9bf287c0-f582-4b83-84c1-15e766218620:66415:1875450488489600:192.168.202.125@tcp:101:touch.0 00000100:00100000:0.0:1788572395.397740:0:66415:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408234300 going to sleep for 11 seconds 00000100:00100000:0.0:1788572395.411090:0:66415:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job touch:9bf287c0-f582-4b83-84c1-15e766218620:66415:1875450488489600:192.168.202.125@tcp:101:touch.0 00010000:00010000:0.0:1788572395.411118:0:66415:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167a00/0x1e260726fbbc0a20 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe9a expref: -99 pid: 66415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572395.411142:0:66415:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167a00/0x1e260726fbbc0a20 lrc: 4/1,0 mode: --/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe9a expref: -99 pid: 66415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572395.411151:0:66415:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167a00/0x1e260726fbbc0a20 lrc: 4/1,0 mode: --/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe9a expref: -99 pid: 66415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572395.411161:0:66415:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167a00/0x1e260726fbbc0a20 lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfe9a expref: -99 pid: 66415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572395.411173:0:66415:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167a00/0x1e260726fbbc0a20 lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe9a expref: -99 pid: 66415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572395.411187:0:66415:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167a00/0x1e260726fbbc0a20 lrc: 5/2,0 mode: PR/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfe9a expref: -99 pid: 66415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572395.411200:0:66415:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167a00/0x1e260726fbbc0a20 lrc: 5/2,0 mode: PR/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfe9a expref: -99 pid: 66415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572395.411875:0:66415:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167a00/0x1e260726fbbc0a20 lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfe9a expref: -99 pid: 66415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572395.411969:0:66415:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff94d408d3f100 x1875450488489600/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 592/688 e 0 to 0 dl 1788572411 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572395.411986:0:66415:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167a00/0x1e260726fbbc0a20 lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe9a expref: -99 pid: 66415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572395.411997:0:66415:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589977617/3 (0 0) 00000080:00200000:0.0:1788572395.412004:0:66415:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572395.415287:0:66415:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x20000a811:0x3:0x0], updating i_size 4096 00000080:00200000:1.0:1788572395.415298:0:66415:0:(namei.c:147:ll_iget()) got inode: [0x20000a811:0x3:0x0](ffff94d4034300c8): rc = 0 00000080:00010000:1.0:1788572395.415315:0:66415:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a811:0x3:0x0](ffff94d4034300c8) for lock 0x1e260726fbbc0a20 00000080:00010000:1.0:1788572395.415333:0:66415:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a811:0x3:0x0](ffff94d4034300c8) 00000080:00010000:1.0:1788572395.415337:0:66415:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a20 from it ffffb68a41f7fac8 00010000:00010000:1.0:1788572395.415345:0:66415:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167a00/0x1e260726fbbc0a20 lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfe9a expref: -99 pid: 66415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572395.415361:0:66415:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167a00/0x1e260726fbbc0a20 lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfe9a expref: -99 pid: 66415 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572395.415407:0:66415:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x20000a811:0x3:0x0](ffff94d4034300c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788572395.415432:0:66415:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x20000a811:0x3:0x0](ffff94d4034300c8), inode mode 41ed mask 3 00000080:00200000:1.0:1788572395.415439:0:66415:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f130b.replay-single, dir=[0x20000a811:0x3:0x0](ffff94d4034300c8), file ffff94d404c60300, open_flags 8941, mode 81b6 opened 0 00000080:00200000:1.0:1788572395.415449:0:66415:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f130b.replay-single, dir=[0x20000a811:0x3:0x0](ffff94d4034300c8), intent=open|creat 00000002:00010000:1.0:1788572395.415478:0:66415:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f130b.replay-single,[0x20000a811:0x5:0x0]) in obj [0x20000a811:0x3:0x0], intent: open|creat flags 0104502 00010000:00010000:1.0:1788572395.415540:0:66415:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a27 lrc: 3/0,1 mode: --/CW res: [0x20000a811:0x3:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 66415 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572395.415550:0:66415:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a27 lrc: 3/0,1 mode: --/CW res: [0x20000a811:0x3:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 66415 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572395.415563:0:66415:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572395.415578:0:66415:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d4049bf800 x1875450488489856/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 752/66408 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572395.415588:0:66415:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a27 lrc: 3/0,1 mode: --/CW res: [0x20000a811:0x3:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 66415 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572395.415610:0:66415:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job touch:9bf287c0-f582-4b83-84c1-15e766218620:66415:1875450488489856:192.168.202.125@tcp:101:touch.0 00000100:00100000:1.0:1788572395.416126:0:66415:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408d81840 going to sleep for 11 seconds 00000100:00100000:1.0:1788572395.431523:0:66415:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job touch:9bf287c0-f582-4b83-84c1-15e766218620:66415:1875450488489856:192.168.202.125@tcp:101:touch.0 00010000:00010000:1.0:1788572395.431551:0:66415:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PW ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a27 lrc: 4/0,1 mode: --/CW res: [0x20000a811:0x3:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfeb6 expref: -99 pid: 66415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572395.431576:0:66415:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a27 lrc: 4/0,1 mode: --/PW res: [0x20000a811:0x5:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfeb6 expref: -99 pid: 66415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572395.431584:0:66415:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a27 lrc: 4/0,1 mode: --/PW res: [0x20000a811:0x5:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfeb6 expref: -99 pid: 66415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572395.431594:0:66415:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a27 lrc: 4/0,1 mode: PW/PW res: [0x20000a811:0x5:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfeb6 expref: -99 pid: 66415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572395.431604:0:66415:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a27 lrc: 4/0,1 mode: PW/PW res: [0x20000a811:0x5:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfeb6 expref: -99 pid: 66415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572395.431618:0:66415:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a27 lrc: 5/0,2 mode: PW/PW res: [0x20000a811:0x5:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfeb6 expref: -99 pid: 66415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572395.431626:0:66415:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a27 lrc: 5/0,2 mode: PW/PW res: [0x20000a811:0x5:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfeb6 expref: -99 pid: 66415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572395.431633:0:66415:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a27 lrc: 4/0,1 mode: PW/PW res: [0x20000a811:0x5:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4dfeb6 expref: -99 pid: 66415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572395.431643:0:66415:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff94d4049bf800 x1875450488489856/t98784247815(98784247815) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 752/824 e 0 to 0 dl 1788572411 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572395.431791:0:66415:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d4049bf800 x1875450488489856/t98784247815(98784247815) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 752/824 e 0 to 0 dl 1788572411 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572395.431871:0:66415:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 216 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a27 lrc: 3/0,1 mode: PW/PW res: [0x20000a811:0x5:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfeb6 expref: -99 pid: 66415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788572395.431884:0:66415:0:(mdc_locks.c:969:mdc_finish_enqueue()) ### DoM lock is returned by: open|creat, size: 0 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a27 lrc: 3/0,1 mode: PW/PW res: [0x20000a811:0x5:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4dfeb6 expref: -99 pid: 66415 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788572395.432021:0:66415:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a27 lrc: 3/0,1 mode: PW/PW res: [0x20000a811:0x5:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4dfeb6 expref: -99 pid: 66415 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572395.432032:0:66415:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 2 res 8589977617/5 (0 0) 00000080:00200000:1.0:1788572395.432040:0:66415:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x20000a811:0x3:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572395.432053:0:66415:0:(namei.c:1628:ll_create_it()) VFS Op:name=f130b.replay-single, dir=[0x20000a811:0x3:0x0](ffff94d4034300c8), intent=open|creat 00000080:00200000:1.0:1788572395.432078:0:66415:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x20000a811:0x5:0x0](ffff94d4034900c8) 00000080:00200000:1.0:1788572395.432190:0:66415:0:(file.c:6826:ll_layout_conf()) [0x20000a811:0x5:0x0]: layout version change: 4294967294 -> 2 00000080:00010000:1.0:1788572395.432220:0:66415:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x20000a811:0x5:0x0](ffff94d4034900c8) 00000080:00010000:1.0:1788572395.432224:0:66415:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a811:0x5:0x0](ffff94d4034900c8) for lock 0x1e260726fbbc0a27 00000080:00010000:1.0:1788572395.432233:0:66415:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a811:0x5:0x0](ffff94d4034900c8) 00000080:00200008:1.0:1788572395.432245:0:66415:0:(file.c:1090:ll_file_open()) START file f130b.replay-single:[0x20000a811:0x5:0x0](ffff94d4034900c8), flags 104501 00000002:00100000:1.0:1788572395.432265:0:66415:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d4049bf800 x1875450488489856/t98784247815(98784247815) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 752/824 e 0 to 0 dl 1788572411 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788572395.432302:0:66415:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a27 from it ffff94d408d4a9c0 00010000:00010000:1.0:1788572395.432306:0:66415:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a27 lrc: 3/0,1 mode: PW/PW res: [0x20000a811:0x5:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4dfeb6 expref: -99 pid: 66415 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572395.432315:0:66415:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a27 lrc: 2/0,0 mode: PW/PW res: [0x20000a811:0x5:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4dfeb6 expref: -99 pid: 66415 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572395.432334:0:66415:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x20000a811:0x5:0x0](ffff94d4034900c8), inode mode 81a4 mask 40 00000080:00200008:1.0:1788572395.432365:0:66415:0:(llite_lib.c:2436:ll_setattr_raw()) START file f130b.replay-single:[0x20000a811:0x5:0x0](ffff94d4034900c8) current size 0, valid attrs 20070, mode 94d4, uid 1106771536, gid -18806, new size 18446744071809989143, atime 1788572395.000000000, mtime 1788572395.000000000, ctime 1788572395.000000000 00000020:00100000:1.0:1788572395.432404:0:66415:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788572395.432411:0:66415:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d4049bd500 x1875450488490112/t0(0) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572395.432430:0:66415:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job touch:9bf287c0-f582-4b83-84c1-15e766218620:66415:1875450488490112:192.168.202.125@tcp:36:touch.0 00000100:00100000:1.0:1788572395.432512:0:66415:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408d81840 going to sleep for 11 seconds 00000100:00100000:3.0:1788572395.461607:0:2396:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.125@tcp (source: 12345-192.168.202.125@tcp) 00000100:00100000:2.0:1788572395.463865:0:42440:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1788572395.464056:0:42440:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875450511440640 00000100:00100000:2.0:1788572395.464066:0:42440:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875450511440640 00000100:00100000:2.0:1788572395.464138:0:42440:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.125@tcp, seq: 90 00000100:00100000:2.0:1788572395.464162:0:42440:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff94d416f32450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:9bf287c0-f582-4b83-84c1-15e766218620+5:66198:x1875450511440640:12345-192.168.202.125@tcp:104: 00010000:00010000:2.0:1788572395.464181:0:42440:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a27 lrc: 2/0,0 mode: PW/PW res: [0x20000a811:0x5:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xec901abfeb4dfeb6 expref: -99 pid: 66415 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572395.464315:0:42440:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788572395.464324:0:42440:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff94d416f32450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:9bf287c0-f582-4b83-84c1-15e766218620+5:66198:x1875450511440640:12345-192.168.202.125@tcp:104: Request processed in 175us (2726us total) trans 0 rc 0/0 00000100:00100000:2.0:1788572395.464332:0:42440:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.125@tcp, seq: 90 00010000:00010000:1.0:1788572395.465579:0:42448:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788572395.465590:0:42448:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a27 lrc: 2/0,0 mode: PW/PW res: [0x20000a811:0x5:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xec901abfeb4dfeb6 expref: -99 pid: 66415 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572395.465601:0:42448:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412582c00 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:1.0:1788572395.465620:0:42448:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a27 lrc: 2/0,0 mode: PW/PW res: [0x20000a811:0x5:0x0].0x0 bits 0x5b/0x2 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xec901abfeb4dfeb6 expref: -99 pid: 66415 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572395.465633:0:42448:0:(ldlm_request.c:1334:ldlm_cli_convert_req()) ### client-side convert ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a27 lrc: 2/0,0 mode: PW/PW res: [0x20000a811:0x5:0x0].0x0 bits 0x5b/0x2 rrc: 2 type: IBT gid 0 flags: 0x420402000000 nid: local remote: 0xec901abfeb4dfeb6 expref: -99 pid: 66415 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1788572395.466023:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488490240:192.168.202.125@tcp:102:ldlm_bl.0 00010000:00010000:1.0:1788572395.466333:0:42448:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a27 lrc: 2/0,0 mode: PW/PW res: [0x20000a811:0x5:0x0].0x0 bits 0x59/0x2 rrc: 2 type: IBT gid 0 flags: 0x10420402000000 nid: local remote: 0xec901abfeb4dfeb6 expref: -99 pid: 66415 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572395.466348:0:42448:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a27 lrc: 2/0,0 mode: PW/PW res: [0x20000a811:0x5:0x0].0x0 bits 0x59/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4dfeb6 expref: -99 pid: 66415 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572395.466357:0:42448:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a27 lrc: 2/0,0 mode: PW/PW res: [0x20000a811:0x5:0x0].0x0 bits 0x59/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4dfeb6 expref: -99 pid: 66415 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572395.466368:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572395.466371:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572395.466373:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572395.478322:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488490240:192.168.202.125@tcp:102:ldlm_bl.0 00000100:00100000:1.0:1788572395.488480:0:66415:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job touch:9bf287c0-f582-4b83-84c1-15e766218620:66415:1875450488490112:192.168.202.125@tcp:36:touch.0 00000080:00200000:1.0:1788572395.488564:0:66415:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x5:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572395.488578:0:66415:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:1.0:1788572395.488602:0:66415:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x0 00000080:00200000:1.0:1788572395.488642:0:66415:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x5:0x0] ignore/verify layout 0/0, layout version 2 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1788572395.488658:0:66415:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file f130b.replay-single:[0x20000a811:0x5:0x0](ffff94d4034900c8) current size 0, valid attrs 100201f0, mode 94d4, uid 1106771536, gid -18806, new size 18446744071809989143, atime 1788572395.000000000, mtime 1788572395.000000000, ctime 1788572395.000000000, rc 0 00000080:00200008:1.0:1788572395.488685:0:66415:0:(file.c:479:ll_file_release()) START file f130b.replay-single:[0x20000a811:0x5:0x0](ffff94d4034900c8), flags 104001 00000002:00100000:1.0:1788572395.488757:0:66415:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff94d4049bf800 x1875450488489856/t98784247815(98784247815) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 752/824 e 0 to 0 dl 1788572411 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572395.488769:0:66415:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572395.488774:0:66415:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d4049bd880 x1875450488490496/t0(0) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/712 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572395.488794:0:66415:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job touch:9bf287c0-f582-4b83-84c1-15e766218620:66415:1875450488490496:192.168.202.125@tcp:35:touch.0 00000100:00100000:1.0:1788572395.488875:0:66415:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408d81840 going to sleep for 11 seconds 00000100:00100000:1.0:1788572395.503514:0:66415:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job touch:9bf287c0-f582-4b83-84c1-15e766218620:66415:1875450488490496:192.168.202.125@tcp:35:touch.0 00000002:00100000:1.0:1788572395.503555:0:66415:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff94d4049bf800 x1875450488489856/t98784247815(98784247815) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 752/824 e 0 to 0 dl 1788572411 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00010000:00010000:3.0:1788572396.165456:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572396.743957:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488490624:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572396.745912:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488490880:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572396.745962:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488491136:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572396.745996:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788572396.746003:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488490752:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572396.746046:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488491008:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572396.755532:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488490752:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572396.755551:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488491008:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572396.755561:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488490624:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572396.755572:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488490880:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572396.755583:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488491136:192.168.202.125@tcp:400:kworker.0 00010000:00010000:2.0:1788572397.192277:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572398.218286:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572399.237460:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572400.265133:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572401.285394:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572401.862131:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488491264:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572401.862157:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488491392:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572401.862250:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488491520:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572401.862278:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488491776:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572401.862357:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488491648:192.168.202.125@tcp:400:kworker.0 00000100:00080000:3.0:1788572401.870955:0:2406:0:(recover.c:205:ptlrpc_request_handle_notconn()) import lustre-MDT0000-mdc-ffff94d405cc3800 of lustre-MDT0000_UUID@192.168.202.125@tcp abruptly disconnected: reconnecting 00000100:00080000:3.0:1788572401.870963:0:2406:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:3.0:1788572401.870970:0:2406:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff94d405cc3800: Connection to lustre-MDT0000 (at 192.168.202.125@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:3.0:1788572401.870978:0:2406:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572401.870982:0:2406:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3509 00000100:00080000:3.0:1788572401.870993:0:2406:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572401.871003:0:2406:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572401.871035:0:2406:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488491904/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572401.871074:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488491392:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572401.871131:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488491520:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572401.871145:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488491776:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572401.871821:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488491904:192.168.202.125@tcp:38:ptlrpcd_01_01.0 00000100:00100000:3.0:1788572401.878344:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488491648:192.168.202.125@tcp:400:kworker.0 00000100:00080000:1.0:1788572401.879133:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572401.879142:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-19) 00000100:00100000:1.0:1788572401.879147:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488491904:192.168.202.125@tcp:38:ptlrpcd_01_01.0 00010000:00010000:3.0:1788572402.310923:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572403.340577:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572404.358340:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572405.382331:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572406.405442:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572406.990798:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572406.990806:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572406.990810:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572406.990821:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3555 00000100:00080000:2.0:1788572406.990831:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: tried all connections, increasing latency to 5s 00000100:00080000:2.0:1788572406.990838:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572406.990848:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572406.990867:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60380 x1875450488492160/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572406.990925:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488492032:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572406.990953:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488492288:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572406.991022:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488492416:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572406.991029:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488492160:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788572407.001150:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488492544:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572407.010522:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488492288:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572407.010544:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488492544:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572407.010580:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488492416:192.168.202.125@tcp:400:kworker.0 00000100:00080000:1.0:1788572407.010722:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572407.010729:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-19) 00000100:00100000:1.0:1788572407.010734:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488492160:192.168.202.125@tcp:38:kworker.0 00010000:00010000:2.0:1788572407.430185:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572408.453400:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572409.477449:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572410.502069:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572411.526008:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572412.102579:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572412.102584:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572412.102588:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572412.102656:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3560 00000100:00080000:3.0:1788572412.102667:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1788572412.102673:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572412.102683:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572412.102701:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488492800/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572412.102735:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405bad000 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00080000:3.0:1788572412.102739:0:51031:0:(import.c:2101:ptlrpc_disconnect_and_idle_import()) lustre-OST0000-osc-ffff94d405cc3800: disconnect after 25s idle 00000100:00080000:3.0:1788572412.102745:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405bab800 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:00080000:3.0:1788572412.102748:0:51031:0:(import.c:2101:ptlrpc_disconnect_and_idle_import()) lustre-OST0001-osc-ffff94d405cc3800: disconnect after 25s idle 00000100:00100000:3.0:1788572412.102773:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488492928:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572412.102950:0:2405:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788572412.102957:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488492672:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572412.103694:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488492800:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572412.103752:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488493056:192.168.202.125@tcp:9:kworker.0 00000100:00100000:1.0:1788572412.103778:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488493184:192.168.202.125@tcp:9:kworker.0 00000100:00100000:3.0:1788572412.119379:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488492928:192.168.202.125@tcp:400:kworker.0 00000100:00080000:3.0:1788572412.119422:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572412.119428:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-19) 00000100:00100000:3.0:1788572412.119431:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488492800:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788572412.119446:0:2402:0:(import.c:2011:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff94d408c55500 x1875450488493056/t0(0) o9->lustre-OST0000-osc-ffff94d405cc3800@192.168.202.125@tcp:28/4 lens 224/224 e 0 to 0 dl 1788572422 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572412.119456:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bad000 lustre-OST0000_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:3.0:1788572412.119460:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488493056:192.168.202.125@tcp:9:kworker.0 00000100:00080000:3.0:1788572412.119470:0:2402:0:(import.c:2011:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff94d408c54e00 x1875450488493184/t0(0) o9->lustre-OST0001-osc-ffff94d405cc3800@192.168.202.125@tcp:28/4 lens 224/224 e 0 to 0 dl 1788572422 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572412.119477:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab800 lustre-OST0001_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:3.0:1788572412.119480:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488493184:192.168.202.125@tcp:9:kworker.0 00010000:00010000:2.0:1788572412.549458:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572413.574037:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572414.599249:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572415.621601:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572416.646575:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:3.0:1788572417.222572:0:2405:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572401/real 1788572401] req@ffff94d408c54a80 x1875450488491264/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572417 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572417.222606:0:2405:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff94d408c54a80 x1875450488491264/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572417 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572417.222615:0:2405:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from FULL to DISCONN 00000100:02020000:3.0:1788572417.222619:0:2405:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.202.125@tcp: Connection to MGS (at 192.168.202.125@tcp) was lost; in progress operations using this service will fail 00000100:00080000:3.0:1788572417.222623:0:2405:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.202.125@tcp for MGC192.168.202.125@tcp not replayable, auto-deactivating 00000100:00080000:3.0:1788572417.222625:0:2405:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:3.0:1788572417.222629:0:2405:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff94d408c54a80 x1875450488491264/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572417 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572417.222638:0:2405:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff94d408c62300 x1875450488492032/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572422 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572417.222663:0:2405:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff94d408c55180 x1875450488492672/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572428 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:3.0:1788572417.222672:0:2405:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:3.0:1788572417.222674:0:2405:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:3.0:1788572417.222675:0:2405:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:3.0:1788572417.222678:0:2405:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00100000:3.0:1788572417.222695:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488492672:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572417.222701:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488491264:192.168.202.125@tcp:400:kworker.0 00000100:00080000:1.0:1788572417.222988:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) e60431de-58b9-4f69-9048-11a3d82ddcdd->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572417.222994:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572417.222996:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572417.223002:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3508 00000100:00080000:1.0:1788572417.223026:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d40490a800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788572417.223057:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488493312/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572417.223092:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572417.223096:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572417.223098:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572417.223102:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3565 00000100:00080000:1.0:1788572417.223107:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: tried all connections, increasing latency to 15s 00000100:00080000:1.0:1788572417.223112:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572417.223121:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572417.223134:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488493440/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572417.223234:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488493568:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572417.223739:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488492032:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572417.224240:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e60431de-58b9-4f69-9048-11a3d82ddcdd:2402:1875450488493312:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788572417.225624:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488493440:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572417.231771:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488493568:192.168.202.125@tcp:400:kworker.0 00000100:00080000:1.0:1788572417.233486:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572417.233498:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-19) 00000100:00100000:1.0:1788572417.233504:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488493440:192.168.202.125@tcp:38:kworker.0 00010000:00010000:3.0:1788572417.670999:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572418.693464:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572419.717442:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572420.741419:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572421.765415:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572422.341508:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) e60431de-58b9-4f69-9048-11a3d82ddcdd->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572422.341520:0:51031:0:(pinger.c:247:ptlrpc_pinger_process_import()) e60431de-58b9-4f69-9048-11a3d82ddcdd->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572422.341524:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572422.341527:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572422.341532:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572422.341543:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3570 00000100:00080000:3.0:1788572422.341554:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572422.341563:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572422.341576:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572422.341612:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488493696/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572422.341766:0:2405:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788572422.341797:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488493824:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572422.341960:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488493696:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788572422.348497:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488493824:192.168.202.125@tcp:400:kworker.0 00000100:00080000:1.0:1788572422.348738:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572422.348745:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-19) 00000100:00100000:1.0:1788572422.348749:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488493696:192.168.202.125@tcp:38:kworker.0 00010000:00010000:1.0:1788572422.789794:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572423.817806:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572424.837719:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572425.861786:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1788572426.437457:0:470:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572426.437472:0:470:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:0.0:1788572426.437491:0:470:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x0 00000080:00200000:0.0:1788572426.437578:0:470:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:1.0:1788572426.886175:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572427.461423:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) e60431de-58b9-4f69-9048-11a3d82ddcdd->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572427.461434:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) e60431de-58b9-4f69-9048-11a3d82ddcdd->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572427.461437:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572427.461441:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572427.461445:0:470:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572427.461455:0:470:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3575 00000100:00080000:3.0:1788572427.461465:0:470:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572427.461474:0:470:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572427.461485:0:470:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572427.461523:0:470:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488493952/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572427.461596:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788572427.461614:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488494080:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572427.462102:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572417/real 1788572417] req@ffff94d4049be300 x1875450488493312/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572427 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572427.462125:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488493312/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572427 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572427.462141:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488493952:192.168.202.125@tcp:38:kworker.0 00000100:00080000:1.0:1788572427.462182:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572427.462224:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572427.462229:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e60431de-58b9-4f69-9048-11a3d82ddcdd:2402:1875450488493312:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788572427.462315:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) e60431de-58b9-4f69-9048-11a3d82ddcdd->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572427.462320:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572427.462324:0:470:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572427.462329:0:470:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3570 00000100:00080000:0.0:1788572427.462336:0:470:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1788572427.462343:0:470:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d40490a800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788572427.462362:0:470:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488494208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572427.462674:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e60431de-58b9-4f69-9048-11a3d82ddcdd:2402:1875450488494208:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788572427.475456:0:2402:0:(import.c:1132:ptlrpc_connect_interpret()) MGC192.168.202.125@tcp: connect to target with instance 0 10000000:01000000:0.0:1788572427.475485:0:2402:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:02000400:0.0:1788572427.475497:0:2402:0:(import.c:1284:ptlrpc_connect_interpret()) Evicted from MGS (at 192.168.202.125@tcp) after server handle changed from 0xec901abfeb4df9ae to 0xec901abfeb4dfee7 00000100:00100000:3.0:1788572427.475499:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488494080:192.168.202.125@tcp:400:kworker.0 00000100:00080000:0.0:1788572427.475510:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from CONNECTING to EVICTED 00000100:00080000:0.0:1788572427.475523:0:2402:0:(import.c:1674:ptlrpc_import_recovery_state_machine()) evicted from MGS@192.168.202.125@tcp; invalidating 00000100:00100000:0.0:1788572427.476889:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e60431de-58b9-4f69-9048-11a3d82ddcdd:2402:1875450488494208:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788572427.477019:0:66638:0:(import.c:1598:ptlrpc_invalidate_import_thread()) thread invalidate import MGC192.168.202.125@tcp to MGS@192.168.202.125@tcp 00000100:00100000:1.0:1788572427.477025:0:66638:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:1.0:1788572427.477029:0:66638:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00010000:00010000:1.0:1788572427.477061:0:66638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.125@tcp lock: ffff94d412167000/0x1e260726fbbc09cc lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110e400000000 nid: local remote: 0xec901abfeb4dfb36 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572427.477072:0:66638:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.125@tcp lock: ffff94d412167000/0x1e260726fbbc09cc lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110f400000000 nid: local remote: 0xec901abfeb4dfb36 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572427.477079:0:66638:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:1.0:1788572427.477095:0:66638:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572427.477105:0:66638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.125@tcp lock: ffff94d412167000/0x1e260726fbbc09cc lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x590f400000000 nid: local remote: 0xec901abfeb4dfb36 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572427.477118:0:66638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.125@tcp lock: ffff94d400030600/0x1e260726fbbc09da lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0xec901abfeb4dfb44 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572427.477125:0:66638:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.125@tcp lock: ffff94d400030600/0x1e260726fbbc09da lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0xec901abfeb4dfb44 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572427.477130:0:66638:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:1.0:1788572427.477134:0:66638:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572427.477137:0:66638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.125@tcp lock: ffff94d400030600/0x1e260726fbbc09da lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0xec901abfeb4dfb44 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572427.477152:0:66638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.125@tcp lock: ffff94d412166c00/0x1e260726fbbc09c5 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0xec901abfeb4dfb2f expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572427.477159:0:66638:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.125@tcp lock: ffff94d412166c00/0x1e260726fbbc09c5 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0xec901abfeb4dfb2f expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572427.477163:0:66638:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788572427.477166:0:66638:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572427.477170:0:66638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.125@tcp lock: ffff94d412166c00/0x1e260726fbbc09c5 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0xec901abfeb4dfb2f expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572427.477185:0:66638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.125@tcp lock: ffff94d412536400/0x1e260726fbbc09d3 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0xec901abfeb4dfb3d expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572427.477192:0:66638:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.125@tcp lock: ffff94d412536400/0x1e260726fbbc09d3 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0xec901abfeb4dfb3d expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572427.477198:0:66638:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 00010000:00010000:1.0:1788572427.477244:0:66638:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572427.477248:0:66638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.125@tcp lock: ffff94d412536400/0x1e260726fbbc09d3 lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0xec901abfeb4dfb3d expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:1.0:1788572427.477263:0:66638:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from EVICTED to RECOVER 00000100:00080000:1.0:1788572427.477268:0:66638:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from RECOVER to FULL 10000000:01000000:1.0:1788572427.477270:0:66638:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:1.0:1788572427.477274:0:66638:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:02000000:1.0:1788572427.477278:0:66638:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) MGC192.168.202.125@tcp: Connection restored to 192.168.202.125@tcp (at 192.168.202.125@tcp) 00000100:00080000:0.0:1788572427.483975:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572427.483990:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-19) 00000100:00100000:0.0:1788572427.483996:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488493952:192.168.202.125@tcp:38:kworker.0 00010000:00010000:2.0:1788572427.909649:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572428.935523:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788572429.254021:0:42451:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788572429.254031:0:42451:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-cliir-ffff94d405cc3800 from 105 10000000:01000000:0.0:1788572429.254038:0:42451:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788572429.254099:0:42451:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412166800/0x1e260726fbbc0a2e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572429.254113:0:42451:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.125@tcp lock: ffff94d412166800/0x1e260726fbbc0a2e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572429.254125:0:42451:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.125@tcp lock: ffff94d412166800/0x1e260726fbbc0a2e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572429.254160:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488494336:192.168.202.125@tcp:101:ll_cfg_requeue.0 00000100:00100000:0.0:1788572429.254293:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4082346c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572429.263596:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488494336:192.168.202.125@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788572429.263626:0:42451:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.125@tcp lock: ffff94d412166800/0x1e260726fbbc0a2e lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e007d expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572429.263640:0:42451:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.125@tcp lock: ffff94d412166800/0x1e260726fbbc0a2e lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xec901abfeb4e007d expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572429.263733:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488494464:192.168.202.125@tcp:256:ll_cfg_requeue.0 00000100:00100000:0.0:1788572429.263815:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4082346c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572429.274668:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488494464:192.168.202.125@tcp:256:ll_cfg_requeue.0 00000100:00080000:0.0:1788572429.274751:0:42451:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff94d405bab000 already not connected (conn 80, was 0): DISCONN 00010000:00080000:0.0:1788572429.274766:0:42451:0:(ldlm_lib.c:85:import_set_conn()) imp ffff94d405bab000@lustre-MDT0000-mdc-ffff94d405cc3800: found existing conn 192.168.202.125@tcp, moved to head 00000100:00080000:0.0:1788572429.274773:0:42451:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572429.274778:0:42451:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 0 00000100:00080000:0.0:1788572429.274788:0:42451:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572429.274800:0:42451:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572429.274831:0:42451:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488494592/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572429.274931:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488494592:192.168.202.125@tcp:38:ll_cfg_requeue.0 10000000:01000000:0.0:1788572429.275860:0:42451:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.202.125@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:0.0:1788572429.275876:0:42451:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412166800/0x1e260726fbbc0a2e lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e007d expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572429.275902:0:42451:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.125@tcp lock: ffff94d412166800/0x1e260726fbbc0a2e lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e007d expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788572429.275914:0:42451:0:(mgc_request.c:611:do_requeue()) updating log lustre-client 10000000:01000000:0.0:1788572429.275917:0:42451:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-client-ffff94d405cc3800 from 52 10000000:01000000:0.0:1788572429.275921:0:42451:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:0.0:1788572429.275957:0:42451:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412166200/0x1e260726fbbc0a35 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572429.275966:0:42451:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.125@tcp lock: ffff94d412166200/0x1e260726fbbc0a35 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572429.275975:0:42451:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.125@tcp lock: ffff94d412166200/0x1e260726fbbc0a35 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572429.275996:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488494720:192.168.202.125@tcp:101:ll_cfg_requeue.0 00000100:00100000:0.0:1788572429.276245:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4082346c0 going to sleep for 11 seconds 00010000:00080000:1.0:1788572429.289725:0:2402:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff94d408d3e680 x1875450488494592/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/416 e 0 to 0 dl 1788572439 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572429.289754:0:2402:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to target with instance 24 00000100:00080000:1.0:1788572429.289775:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:1.0:1788572429.289783:0:2402:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:1.0:1788572429.289786:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 98784247812 gen 4 00000100:00100000:1.0:1788572429.289792:0:2402:0:(client.c:3156:ptlrpc_free_committed()) @@@ stopping search req@ffff94d408c54380 x1875450488488192/t98784247813(98784247813) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572403 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572429.289806:0:2402:0:(recover.c:53:ptlrpc_replay_next()) import ffff94d405bab000 from lustre-MDT0000_UUID committed 98784247812 last 0 00000100:00080000:1.0:1788572429.289814:0:2402:0:(client.c:3540:ptlrpc_replay_req()) @@@ REPLAY req@ffff94d408c54380 x1875450488488192/t98784247813(98784247813) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572403 ref 1 fl New:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572429.289826:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488494592:192.168.202.125@tcp:38:ll_cfg_requeue.0 00000100:00100000:1.0:1788572429.289858:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488488192:192.168.202.125@tcp:36:mcreate.0 00000100:00100000:0.0:1788572429.290543:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488494720:192.168.202.125@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788572429.290566:0:42451:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.125@tcp lock: ffff94d412166200/0x1e260726fbbc0a35 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0092 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572429.290579:0:42451:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.125@tcp lock: ffff94d412166200/0x1e260726fbbc0a35 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xec901abfeb4e0092 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572429.290650:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488494848:192.168.202.125@tcp:501:ll_cfg_requeue.0 00000100:00100000:0.0:1788572429.290695:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4082346c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572429.295806:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488494848:192.168.202.125@tcp:501:ll_cfg_requeue.0 00000100:00100000:0.0:1788572429.295869:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488494976:192.168.202.125@tcp:503:ll_cfg_requeue.0 00000100:00100000:0.0:1788572429.295983:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4082346c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572429.299438:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488494976:192.168.202.125@tcp:503:ll_cfg_requeue.0 00000040:00080000:0.0:1788572429.305392:0:66649:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 64768 count 51 00000020:01000000:3.0:1788572429.319006:0:42451:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 52-51 (rc=0) 10000000:01000000:3.0:1788572429.319049:0:42451:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.202.125@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:3.0:1788572429.319062:0:42451:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412166200/0x1e260726fbbc0a35 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0092 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572429.319074:0:42451:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.125@tcp lock: ffff94d412166200/0x1e260726fbbc0a35 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0092 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572429.319107:0:42451:0:(mgc_request.c:611:do_requeue()) updating log params 10000000:01000000:3.0:1788572429.319110:0:42451:0:(mgc_request.c:1801:mgc_process_log()) Process log params-ffff94d405cc3800 from 7 10000000:01000000:3.0:1788572429.319113:0:42451:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:3.0:1788572429.319153:0:42451:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d400031c00/0x1e260726fbbc0a3c lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572429.319161:0:42451:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.125@tcp lock: ffff94d400031c00/0x1e260726fbbc0a3c lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572429.319187:0:42451:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.125@tcp lock: ffff94d400031c00/0x1e260726fbbc0a3c lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572429.319245:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488495104:192.168.202.125@tcp:101:ll_cfg_requeue.0 00000100:00100000:3.0:1788572429.319341:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4128426c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572429.322858:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488495104:192.168.202.125@tcp:101:ll_cfg_requeue.0 00010000:00010000:3.0:1788572429.322899:0:42451:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.125@tcp lock: ffff94d400031c00/0x1e260726fbbc0a3c lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e00b5 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572429.322913:0:42451:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.125@tcp lock: ffff94d400031c00/0x1e260726fbbc0a3c lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xec901abfeb4e00b5 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572429.322992:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488495232:192.168.202.125@tcp:501:ll_cfg_requeue.0 00000100:00100000:3.0:1788572429.323193:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4128426c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572429.343870:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488495232:192.168.202.125@tcp:501:ll_cfg_requeue.0 00000100:00100000:3.0:1788572429.343955:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488495360:192.168.202.125@tcp:503:ll_cfg_requeue.0 00000100:00100000:3.0:1788572429.344075:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4128426c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572429.353050:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488495360:192.168.202.125@tcp:503:ll_cfg_requeue.0 00000040:00080000:1.0:1788572429.363463:0:66650:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 64768 count 7 00000020:01000000:3.0:1788572429.368620:0:42451:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 7-6 (rc=0) 10000000:01000000:3.0:1788572429.368654:0:42451:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.202.125@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:3.0:1788572429.368668:0:42451:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d400031c00/0x1e260726fbbc0a3c lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e00b5 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572429.368681:0:42451:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.125@tcp lock: ffff94d400031c00/0x1e260726fbbc0a3c lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e00b5 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572429.368692:0:42451:0:(mgc_request.c:611:do_requeue()) updating log lustre-sptlrpc 10000000:01000000:3.0:1788572429.368695:0:42451:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-sptlrpc-ffff94d4113f0000 from 1 10000000:01000000:3.0:1788572429.368698:0:42451:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:3.0:1788572429.368744:0:42451:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d400031600/0x1e260726fbbc0a43 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572429.368752:0:42451:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.125@tcp lock: ffff94d400031600/0x1e260726fbbc0a43 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572429.368762:0:42451:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.125@tcp lock: ffff94d400031600/0x1e260726fbbc0a43 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572429.368795:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488495488:192.168.202.125@tcp:101:ll_cfg_requeue.0 00000100:00100000:3.0:1788572429.368989:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4128426c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572429.384056:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488495488:192.168.202.125@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788572429.384086:0:42451:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.125@tcp lock: ffff94d400031600/0x1e260726fbbc0a43 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e00df expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572429.384099:0:42451:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.125@tcp lock: ffff94d400031600/0x1e260726fbbc0a43 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xec901abfeb4e00df expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572429.384182:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488495616:192.168.202.125@tcp:501:ll_cfg_requeue.0 00000100:00100000:2.0:1788572429.384275:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405101a80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572429.393866:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488495616:192.168.202.125@tcp:501:ll_cfg_requeue.0 10000000:01000000:2.0:1788572429.393930:0:42451:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.202.125@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:2.0:1788572429.393943:0:42451:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d400031600/0x1e260726fbbc0a43 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e00df expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572429.393959:0:42451:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.125@tcp lock: ffff94d400031600/0x1e260726fbbc0a43 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e00df expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572429.968391:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572430.984991:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572432.006816:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572432.588974:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 1 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572432.588982:0:51031:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:3.0:1788572432.589019:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488495872:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572432.589128:0:2405:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788572432.589135:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488495744:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572432.595489:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488495872:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572432.598810:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488495744:192.168.202.125@tcp:400:kworker.0 00010000:00010000:1.0:1788572433.031404:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572433.473006:0:2402:0:(client.c:3425:ptlrpc_replay_interpret()) @@@ got reply req@ffff94d408c54380 x1875450488488192/t98784247813(98784247813) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572445 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572433.473025:0:2402:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:1.0:1788572433.473028:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 98784247812 gen 4 00000100:00100000:1.0:1788572433.473034:0:2402:0:(client.c:3156:ptlrpc_free_committed()) @@@ stopping search req@ffff94d408c54380 x1875450488488192/t98784247813(98784247813) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572445 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572433.473042:0:2402:0:(recover.c:53:ptlrpc_replay_next()) import ffff94d405bab000 from lustre-MDT0000_UUID committed 98784247812 last 98784247813 00000100:00080000:1.0:1788572433.473050:0:2402:0:(client.c:3540:ptlrpc_replay_req()) @@@ REPLAY req@ffff94d408c63800 x1875450488488704/t98784247814(98784247814) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572403 ref 1 fl New:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572433.473063:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488488192:192.168.202.125@tcp:36:mcreate.0 00000100:00100000:1.0:1788572433.473086:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488488704:192.168.202.125@tcp:36:rm.0 00000100:00080000:1.0:1788572433.484538:0:2402:0:(client.c:3425:ptlrpc_replay_interpret()) @@@ got reply req@ffff94d408c63800 x1875450488488704/t98784247814(98784247814) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572449 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572433.484571:0:2402:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:1.0:1788572433.484574:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 98784247812 gen 4 00000100:00100000:1.0:1788572433.484581:0:2402:0:(client.c:3156:ptlrpc_free_committed()) @@@ stopping search req@ffff94d408c54380 x1875450488488192/t98784247813(98784247813) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572445 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572433.484589:0:2402:0:(recover.c:53:ptlrpc_replay_next()) import ffff94d405bab000 from lustre-MDT0000_UUID committed 98784247812 last 98784247814 00000100:00080000:1.0:1788572433.484597:0:2402:0:(client.c:3540:ptlrpc_replay_req()) @@@ REPLAY req@ffff94d4049bf800 x1875450488489856/t98784247815(98784247815) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 752/824 e 0 to 0 dl 1788572411 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788572433.484609:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488488704:192.168.202.125@tcp:36:rm.0 00000100:00100000:1.0:1788572433.484631:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488489856:192.168.202.125@tcp:101:touch.0 00000100:00100000:1.0:1788572433.492326:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 98784247813 gen 4 00000100:00100000:1.0:1788572433.492371:0:2402:0:(client.c:3156:ptlrpc_free_committed()) @@@ stopping search req@ffff94d408c63800 x1875450488488704/t98784247814(98784247814) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572449 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572433.492389:0:2402:0:(client.c:3425:ptlrpc_replay_interpret()) @@@ got reply req@ffff94d4049bf800 x1875450488489856/t98784247815(98784247815) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 752/608 e 0 to 0 dl 1788572449 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'touch.0' uid:0 gid:0 projid:0 00000002:00080000:1.0:1788572433.492405:0:2402:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff94d4049bd880 x1875450488490496/t98784247817(98784247817) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/456 e 0 to 0 dl 1788572411 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000100:00020000:1.0:1788572433.492413:0:2402:0:(client.c:3435:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff94d4049bf800 x1875450488489856/t98784247815(98784247815) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 752/608 e 0 to 0 dl 1788572449 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'touch.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788572433.520783:0:2402:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:1.0:1788572433.520793:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 98784247813 gen 4 00000100:00100000:1.0:1788572433.520814:0:2402:0:(client.c:3156:ptlrpc_free_committed()) @@@ stopping search req@ffff94d408c63800 x1875450488488704/t98784247814(98784247814) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572449 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572433.520829:0:2402:0:(recover.c:53:ptlrpc_replay_next()) import ffff94d405bab000 from lustre-MDT0000_UUID committed 98784247813 last 98784247815 00000100:00080000:1.0:1788572433.520839:0:2402:0:(client.c:3540:ptlrpc_replay_req()) @@@ REPLAY req@ffff94d4049bd500 x1875450488490112/t98784247816(98784247816) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 488/456 e 0 to 0 dl 1788572411 ref 1 fl New:RQU/204/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572433.520856:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488489856:192.168.202.125@tcp:101:touch.0 00000100:00100000:1.0:1788572433.520886:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488490112:192.168.202.125@tcp:36:touch.0 00000100:00080000:1.0:1788572433.534767:0:2402:0:(client.c:3425:ptlrpc_replay_interpret()) @@@ got reply req@ffff94d4049bd500 x1875450488490112/t98784247816(98784247816) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 488/456 e 0 to 0 dl 1788572449 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572433.534782:0:2402:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:1.0:1788572433.534785:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 98784247813 gen 4 00000100:00100000:1.0:1788572433.534790:0:2402:0:(client.c:3156:ptlrpc_free_committed()) @@@ stopping search req@ffff94d408c63800 x1875450488488704/t98784247814(98784247814) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572449 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572433.534798:0:2402:0:(recover.c:53:ptlrpc_replay_next()) import ffff94d405bab000 from lustre-MDT0000_UUID committed 98784247813 last 98784247816 00000100:00080000:1.0:1788572433.534806:0:2402:0:(client.c:3540:ptlrpc_replay_req()) @@@ REPLAY req@ffff94d4049bd880 x1875450488490496/t98784247817(98784247817) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/456 e 0 to 0 dl 1788572411 ref 2 fl New:RQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788572433.534818:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488490112:192.168.202.125@tcp:36:touch.0 00000100:00100000:1.0:1788572433.534842:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488490496:192.168.202.125@tcp:35:touch.0 00000100:00080000:1.0:1788572433.538963:0:2402:0:(client.c:3425:ptlrpc_replay_interpret()) @@@ got reply req@ffff94d4049bd880 x1875450488490496/t98784247817(98784247817) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/456 e 0 to 0 dl 1788572449 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788572433.538979:0:2402:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:1.0:1788572433.538982:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 98784247813 gen 4 00000100:00100000:1.0:1788572433.538989:0:2402:0:(client.c:3156:ptlrpc_free_committed()) @@@ stopping search req@ffff94d408c63800 x1875450488488704/t98784247814(98784247814) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572449 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572433.539003:0:2402:0:(recover.c:53:ptlrpc_replay_next()) import ffff94d405bab000 from lustre-MDT0000_UUID committed 98784247813 last 98784247817 00000100:00080000:1.0:1788572433.539009:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:1.0:1788572433.539721:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488490496:192.168.202.125@tcp:35:touch.0 00010000:00080000:2.0:1788572433.539773:0:66687:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff94d405cc3800 to lustre-MDT0000_UUID@192.168.202.125@tcp 00010000:00010000:2.0:1788572433.539778:0:66687:0:(ldlm_request.c:2888:ldlm_cancel_unused_locks_for_replay()) Dropping as many unused locks as possible before replay for namespace lustre-MDT0000-mdc-ffff94d405cc3800 (3) 00000080:00200000:2.0:1788572433.539882:0:66687:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572433.539906:0:66687:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572433.539940:0:66687:0:(vvp_object.c:117:vvp_conf_set()) [0x20000a811:0x5:0x0]: losing layout lock 00010000:00010000:2.0:1788572433.539991:0:66687:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a27 lrc: 0/0,0 mode: --/PW res: [0x20000a811:0x5:0x0].0x0 bits 0x59/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0xec901abfeb4dfeb6 expref: -99 pid: 66415 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572433.540022:0:66687:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167a00/0x1e260726fbbc0a20 lrc: 0/0,0 mode: --/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0xec901abfeb4dfe9a expref: -99 pid: 66415 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572433.540040:0:66687:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0a19 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0xec901abfeb4dfe7e expref: -99 pid: 66415 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572433.540050:0:66687:0:(ldlm_request.c:2901:ldlm_cancel_unused_locks_for_replay()) Canceled 0 unused locks from namespace lustre-MDT0000-mdc-ffff94d405cc3800 00000100:00080000:2.0:1788572433.541019:0:66687:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to REPLAY_WAIT 00000100:00100000:1.0:1788572433.541124:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488496000:192.168.202.125@tcp:400:ldlm_lock_repla.0 00000100:00100000:0.0:1788572433.569949:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 98784247817 gen 4 00000100:00080000:0.0:1788572433.569999:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from REPLAY_WAIT to RECOVER 00000100:00080000:0.0:1788572433.570015:0:2402:0:(client.c:3234:ptlrpc_resend_req()) @@@ going to resend req@ffff94d408c62300 x1875450488496000/t0(0) o400->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572449 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572433.570025:0:2402:0:(client.c:3243:ptlrpc_resend_req()) @@@ it has reply, so skip it req@ffff94d408c62300 x1875450488496000/t0(0) o400->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572449 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572433.570032:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from RECOVER to FULL 00000100:02000000:0.0:1788572433.570516:0:2402:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) lustre-MDT0000-mdc-ffff94d405cc3800: Connection restored to 192.168.202.125@tcp (at 192.168.202.125@tcp) 00000100:00100000:0.0:1788572433.570523:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488496000:192.168.202.125@tcp:400:ldlm_lock_repla.0 00000100:00100000:1.0:1788572433.605477:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488496256:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572433.605587:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:1.0:1788572433.605594:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488496128:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572433.605629:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488496384:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572433.612268:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488496256:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572433.612864:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488496128:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572433.612876:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488496384:192.168.202.125@tcp:400:kworker.0 00010000:00010000:2.0:1788572434.053429:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572435.077419:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572436.110709:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572437.126470:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:3.0:1788572437.800029:0:66741:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572437.800043:0:66741:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:3.0:1788572437.800066:0:66741:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572437.800128:0:66741:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 66741 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572437.800139:0:66741:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 66741 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572437.800151:0:66741:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 66741 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572437.800180:0:66741:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:66741:1875450488496512:192.168.202.125@tcp:101:lfs.0 00000100:00100000:3.0:1788572437.801421:0:66741:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d412842e40 going to sleep for 11 seconds 00000100:00100000:0.0:1788572437.804177:0:66741:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:66741:1875450488496512:192.168.202.125@tcp:101:lfs.0 00010000:00010000:0.0:1788572437.804243:0:66741:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572437.804256:0:66741:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572437.804267:0:66741:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572437.804283:0:66741:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572437.804296:0:66741:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572437.804304:0:66741:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572437.804311:0:66741:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572437.804324:0:66741:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408c57b80 x1875450488496512/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572453 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572437.804339:0:66741:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572437.804352:0:66741:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788572437.804370:0:66741:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788572437.804400:0:66741:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1788572437.804406:0:66741:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0a4a 00000080:00010000:0.0:1788572437.804411:0:66741:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572437.804414:0:66741:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a4a from it ffffb68a42327cb8 00010000:00010000:0.0:1788572437.804417:0:66741:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572437.804429:0:66741:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1788572437.805279:0:66741:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1788572437.805288:0:66741:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:0.0:1788572437.805307:0:66741:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572437.805319:0:66741:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572437.805329:0:66741:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572437.805337:0:66741:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572437.805343:0:66741:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572437.805346:0:66741:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0a4a 00000080:00010000:0.0:1788572437.805351:0:66741:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a4a from it ffffb68a42327cb8 00010000:00010000:0.0:1788572437.805368:0:66741:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572437.805376:0:66741:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788572437.805493:0:66741:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:0.0:1788572437.805503:0:66741:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572437.805510:0:66741:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572437.805524:0:66741:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788572437.805532:0:66741:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572437.805536:0:66741:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572437.805538:0:66741:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0a4a 00000080:00010000:0.0:1788572437.805542:0:66741:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a4a from it ffffb68a42327bb0 00010000:00010000:0.0:1788572437.805544:0:66741:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572437.805551:0:66741:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788572437.805559:0:66741:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 44 00000080:00200008:0.0:1788572437.805566:0:66741:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 100000 00000080:00200080:0.0:1788572437.805796:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:0.0:1788572437.805851:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00800000:00000004:0.0:1788572437.805858:0:66741:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0000-mdc-ffff94d405cc3800: age 3540, max_age 3590 00000100:00100000:0.0:1788572437.805898:0:66741:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:66741:1875450488496640:192.168.202.125@tcp:41:lfs.0 00000100:00100000:0.0:1788572437.805991:0:66741:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408234e40 going to sleep for 11 seconds 00000100:00100000:1.0:1788572437.812571:0:66741:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:66741:1875450488496640:192.168.202.125@tcp:41:lfs.0 00800000:00000004:1.0:1788572437.812603:0:66741:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0000-mdc-ffff94d405cc3800: update ffff94d412909e10 cache blocks 317131/353529 objects 1023604/1024000 00000080:00200080:1.0:1788572437.812671:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.812687:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00800000:00000004:1.0:1788572437.812696:0:66741:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0001-mdc-ffff94d405cc3800: age 3540, max_age 3590 00000100:00100000:1.0:1788572437.812734:0:66741:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:66741:1875450488496768:192.168.202.125@tcp:41:lfs.0 00000100:00100000:1.0:1788572437.812819:0:66741:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408d810c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572437.815593:0:66741:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:66741:1875450488496768:192.168.202.125@tcp:41:lfs.0 00800000:00000004:1.0:1788572437.815622:0:66741:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0001-mdc-ffff94d405cc3800: update ffff94d407db9e10 cache blocks 317207/353529 objects 1023684/1024000 00000080:00200080:1.0:1788572437.815751:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.815767:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.815774:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.815779:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.815783:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.815787:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.815791:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.815794:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.815799:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.815802:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.815806:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.815810:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.815814:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.815817:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.815821:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.815825:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.815828:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.815832:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.815836:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.815839:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.815843:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.815848:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.815852:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.815855:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.815859:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.815864:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.815867:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.815871:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.815875:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.815879:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.815883:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.815886:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.815890:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.815893:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.815897:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.815901:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.815904:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.815908:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.815912:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.815916:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.815919:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.815923:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.815927:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.815931:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816000:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816008:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816014:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816018:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816022:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816026:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816030:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816033:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816037:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816041:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816044:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816048:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816052:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816055:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816059:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816063:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816066:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816070:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816074:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816078:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816081:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816085:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816089:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816092:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816096:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816099:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816103:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816107:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816111:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816115:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816119:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816123:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816126:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816130:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816134:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816137:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816141:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816144:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816147:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816150:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816154:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816158:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816161:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816164:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816168:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816172:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816176:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816179:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816183:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816186:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816190:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816194:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816198:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816408:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816416:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816420:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816424:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816427:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816431:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816435:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816439:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816442:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816446:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816450:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816454:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816457:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816461:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816465:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816469:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816472:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816476:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816480:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816484:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816487:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816491:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816495:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816499:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816503:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816507:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816510:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816514:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00800000:00000080:1.0:1788572437.816517:0:66741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.816521:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.816527:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00020000:00000004:1.0:1788572437.816535:0:66741:0:(obd_class.h:1091:obd_statfs()) lustre-OST0000-osc-ffff94d405cc3800: age 3540, max_age 3590 00000100:00080000:1.0:1788572437.816547:0:66741:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3519 00000100:00080000:1.0:1788572437.816557:0:66741:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d405cc3800: import ffff94d405bad000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572437.816567:0:66741:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8437760, lost: 0. 00000100:00100000:1.0:1788572437.816589:0:66741:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488496896/t0(0) o8->lustre-OST0000-osc-ffff94d405cc3800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572437.816712:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488496896:192.168.202.125@tcp:8:lfs.0 00000100:00080000:1.0:1788572437.817078:0:66741:0:(client.c:1832:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff94d4049be680 x1875450488497024/t0(0) o13->lustre-OST0000-osc-ffff94d405cc3800@192.168.202.125@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572437.817088:0:66741:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408d810c0 going to sleep for 0 seconds 00000100:00100000:2.0:1788572437.820175:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-OST0000-osc-ffff94d405cc3800: committing for last_committed 4294968859 gen 15 00000100:00080000:2.0:1788572437.820196:0:2402:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff94d405cc3800: connect to target with instance 1 00000100:00080000:2.0:1788572437.820250:0:2402:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff94d405cc3800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:2.0:1788572437.820263:0:2402:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:2.0:1788572437.820267:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bad000 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00000080:00000004:2.0:1788572437.820276:0:2402:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:2.0:1788572437.820282:0:2402:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:2.0:1788572437.820296:0:2402:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000002db80fc5): req@ffff94d4049be680 x1875450488497024/t0(0) o13->lustre-OST0000-osc-ffff94d405cc3800@192.168.202.125@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572437.820312:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488496896:192.168.202.125@tcp:8:lfs.0 00000100:00100000:1.0:1788572437.820555:0:66741:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408d810c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572437.824032:0:66741:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:66741:1875450488497024:192.168.202.125@tcp:13:lfs.0 00020000:00000004:1.0:1788572437.824062:0:66741:0:(obd_class.h:1117:obd_statfs()) lustre-OST0000-osc-ffff94d405cc3800: update ffff94d407dbd190 cache blocks 901369/958279 objects 261733/262144 00000080:00200080:1.0:1788572437.824128:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.824145:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00020000:00000004:1.0:1788572437.824152:0:66741:0:(obd_class.h:1091:obd_statfs()) lustre-OST0001-osc-ffff94d405cc3800: age 3540, max_age 3590 00000100:00080000:1.0:1788572437.824163:0:66741:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3519 00000100:00080000:1.0:1788572437.824173:0:66741:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d405cc3800: import ffff94d405bab800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572437.824184:0:66741:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8437760, lost: 0. 00000100:00100000:1.0:1788572437.824425:0:66741:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488497152/t0(0) o8->lustre-OST0001-osc-ffff94d405cc3800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572437.824485:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488497152:192.168.202.125@tcp:8:lfs.0 00000100:00080000:1.0:1788572437.825024:0:66741:0:(client.c:1832:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff94d4049be680 x1875450488497280/t0(0) o13->lustre-OST0001-osc-ffff94d405cc3800@192.168.202.125@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572437.825034:0:66741:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408d810c0 going to sleep for 0 seconds 00000100:00100000:2.0:1788572437.828064:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-OST0001-osc-ffff94d405cc3800: committing for last_committed 0 gen 16 00000100:00080000:2.0:1788572437.828081:0:2402:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff94d405cc3800: connect to target with instance 1 00000100:00080000:2.0:1788572437.828091:0:2402:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff94d405cc3800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:2.0:1788572437.828104:0:2402:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:2.0:1788572437.828108:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab800 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00000080:00000004:2.0:1788572437.828117:0:2402:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:2.0:1788572437.828122:0:2402:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:2.0:1788572437.828138:0:2402:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000002db80fc5): req@ffff94d4049be680 x1875450488497280/t0(0) o13->lustre-OST0001-osc-ffff94d405cc3800@192.168.202.125@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572437.828504:0:66741:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408d810c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572437.830833:0:66741:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:66741:1875450488497280:192.168.202.125@tcp:13:lfs.0 00020000:00000004:1.0:1788572437.830861:0:66741:0:(obd_class.h:1117:obd_statfs()) lustre-OST0001-osc-ffff94d405cc3800: update ffff94d407dbb7d0 cache blocks 901369/958279 objects 261733/262144 00000080:00200080:1.0:1788572437.830932:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.830959:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.830967:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.830971:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.830975:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.830978:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.830982:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.830986:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.830989:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.830993:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.830997:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831001:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831004:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831008:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831011:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831015:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831019:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831022:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831026:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831030:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831034:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831038:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831042:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831045:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831050:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831053:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831057:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831060:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831064:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831067:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831071:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831074:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831078:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831082:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831085:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831089:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831093:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831096:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831099:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831102:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831105:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831109:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831113:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831116:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831120:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831123:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831126:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831129:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831133:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831137:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831141:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831144:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831148:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831151:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831154:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831158:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831161:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831164:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831168:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831171:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831175:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831178:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831182:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831185:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831189:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831192:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831196:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831200:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831670:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831678:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831682:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831686:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831689:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831693:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831697:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831700:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831704:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831707:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831711:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831715:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831719:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831722:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831726:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831729:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831733:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831736:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831740:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831743:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831747:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831750:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831754:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831757:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831761:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831764:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831768:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831771:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831775:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831778:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831782:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831785:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831789:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831791:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831795:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831798:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831801:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831804:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831808:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831811:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831814:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831817:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831821:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831824:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831828:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831832:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831836:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831839:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831843:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831846:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831850:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831853:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831856:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831859:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831863:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831866:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200080:1.0:1788572437.831870:0:66741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff679ee5b0 00020000:00000080:1.0:1788572437.831873:0:66741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55000 uarg=0000000000000000 00000080:00200008:1.0:1788572437.831886:0:66741:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 100000 00000100:00100000:2.0:1788572437.836408:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488497152:192.168.202.125@tcp:8:lfs.0 00010000:00010000:3.0:1788572438.155055:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572438.725931:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488497408:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572438.727788:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488497664:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572438.727880:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488497920:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572438.727913:0:2405:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1788572438.727918:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488497536:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572438.727941:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488497792:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572438.734871:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488497408:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572438.745465:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488497792:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572438.745493:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488497664:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572438.745504:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488497920:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572438.745542:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488497536:192.168.202.125@tcp:400:kworker.0 00010000:00010000:1.0:1788572439.173494:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572440.198361:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572441.221831:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572442.247418:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572443.269590:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572443.846985:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488498176:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572443.847077:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488498432:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572443.847114:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:1.0:1788572443.847120:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488498048:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572443.847152:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488498304:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572443.847174:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488498560:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572443.871057:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488498048:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572443.871081:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488498304:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572443.871092:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488498560:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572443.871102:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488498176:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572443.871112:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488498432:192.168.202.125@tcp:400:kworker.0 00000001:02000400:2.0:1788572444.237655:0:67022:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 00010000:00010000:3.0:1788572444.314478:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572445.317375:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572446.342080:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1788572446.465771:0:67170:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 00010000:00010000:3.0:1788572447.366720:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572448.390987:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:3.0:1788572448.689638:0:67208:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572448.689654:0:67208:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:3.0:1788572448.689680:0:67208:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572448.689701:0:67208:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572448.689715:0:67208:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788572448.689723:0:67208:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572448.689732:0:67208:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:3.0:1788572448.689736:0:67208:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0a4a 00000080:00010000:3.0:1788572448.689745:0:67208:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a4a from it ffffb68a41ff7cb8 00010000:00010000:3.0:1788572448.689749:0:67208:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572448.689758:0:67208:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:3.0:1788572448.690359:0:67208:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788572448.690366:0:67208:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:3.0:1788572448.690378:0:67208:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572448.690387:0:67208:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788572448.690397:0:67208:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1788572448.690404:0:67208:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572448.690409:0:67208:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:3.0:1788572448.690412:0:67208:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0a4a 00000080:00010000:3.0:1788572448.696789:0:67208:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a4a from it ffffb68a41ff7cb8 00010000:00010000:3.0:1788572448.696796:0:67208:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788572448.696946:0:67208:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788572448.697410:0:67208:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:3.0:1788572448.697444:0:67208:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572448.697780:0:67208:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788572448.697795:0:67208:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788572448.697856:0:67208:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572448.697866:0:67208:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:3.0:1788572448.697869:0:67208:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0a4a 00000080:00010000:3.0:1788572448.697875:0:67208:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a4a from it ffffb68a41ff7bb0 00010000:00010000:3.0:1788572448.697879:0:67208:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788572448.697886:0:67208:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1788572448.697899:0:67208:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 44 00000080:00200008:3.0:1788572448.697909:0:67208:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 100000 00000080:00200080:3.0:1788572448.698189:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:3.0:1788572448.698549:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f30000 uarg=0000000000000000 00800000:00000004:3.0:1788572448.698558:0:67208:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0000-mdc-ffff94d405cc3800: age 3591, max_age 3601 00000100:00100000:3.0:1788572448.698600:0:67208:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67208:1875450488498688:192.168.202.125@tcp:41:lfs.0 00000100:00100000:3.0:1788572448.698939:0:67208:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d412842780 going to sleep for 11 seconds 00000100:00100000:1.0:1788572448.708762:0:67208:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67208:1875450488498688:192.168.202.125@tcp:41:lfs.0 00800000:00000004:1.0:1788572448.708798:0:67208:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0000-mdc-ffff94d405cc3800: update ffff94d412909e10 cache blocks 317131/353529 objects 1023604/1024000 00000080:00200080:1.0:1788572448.708893:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.708913:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00800000:00000004:1.0:1788572448.708922:0:67208:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0001-mdc-ffff94d405cc3800: age 3591, max_age 3601 00000100:00100000:1.0:1788572448.708967:0:67208:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67208:1875450488498816:192.168.202.125@tcp:41:lfs.0 00000100:00100000:1.0:1788572448.709063:0:67208:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4110f9900 going to sleep for 11 seconds 00000100:00100000:1.0:1788572448.714584:0:67208:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67208:1875450488498816:192.168.202.125@tcp:41:lfs.0 00800000:00000004:1.0:1788572448.714618:0:67208:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0001-mdc-ffff94d405cc3800: update ffff94d407db9e10 cache blocks 317207/353529 objects 1023684/1024000 00000080:00200080:1.0:1788572448.714686:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715077:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715089:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715096:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715101:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715105:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715110:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715113:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715117:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715121:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715124:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715127:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715131:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715134:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715139:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715142:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715147:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715150:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715154:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715158:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715162:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715166:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715170:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715174:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715178:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715181:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715185:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715189:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715193:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715196:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715229:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715237:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715243:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715246:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715250:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715254:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715258:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715261:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715265:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715268:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715272:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715275:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715279:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715282:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715286:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715290:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715293:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715297:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715301:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715304:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715308:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715425:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715433:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715437:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715441:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715444:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715448:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715520:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715529:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715535:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715539:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715543:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715547:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715551:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715555:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715559:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715563:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715566:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715570:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715573:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715578:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715581:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715586:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715589:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715593:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715597:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715600:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715603:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715607:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715610:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715614:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715617:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715621:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715624:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715627:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715630:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715633:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715636:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715640:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715643:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715646:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715649:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715653:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715656:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715659:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715663:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715666:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715669:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715673:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715676:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715681:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715685:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715689:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715693:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715696:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715771:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715780:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715785:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715789:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715792:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715796:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715799:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715803:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715867:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715874:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715878:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715882:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715886:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715889:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715893:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715897:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715900:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715904:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715907:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715911:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00800000:00000080:1.0:1788572448.715915:0:67208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.715919:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.715924:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00020000:00000004:1.0:1788572448.715931:0:67208:0:(obd_class.h:1091:obd_statfs()) lustre-OST0000-osc-ffff94d405cc3800: age 3591, max_age 3601 00000100:00100000:1.0:1788572448.715972:0:67208:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67208:1875450488498944:192.168.202.125@tcp:13:lfs.0 00000100:00100000:1.0:1788572448.717155:0:67208:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4110f9900 going to sleep for 11 seconds 00000100:00100000:1.0:1788572448.725155:0:67208:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67208:1875450488498944:192.168.202.125@tcp:13:lfs.0 00020000:00000004:1.0:1788572448.725192:0:67208:0:(obd_class.h:1117:obd_statfs()) lustre-OST0000-osc-ffff94d405cc3800: update ffff94d407dbd190 cache blocks 901369/958279 objects 261733/262144 00000080:00200080:1.0:1788572448.725307:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.725336:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00020000:00000004:1.0:1788572448.725346:0:67208:0:(obd_class.h:1091:obd_statfs()) lustre-OST0001-osc-ffff94d405cc3800: age 3591, max_age 3601 00000100:00100000:1.0:1788572448.725386:0:67208:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67208:1875450488499072:192.168.202.125@tcp:13:lfs.0 00000100:00100000:1.0:1788572448.725926:0:67208:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4110f9900 going to sleep for 11 seconds 00000100:00100000:1.0:1788572448.730601:0:67208:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67208:1875450488499072:192.168.202.125@tcp:13:lfs.0 00020000:00000004:1.0:1788572448.730634:0:67208:0:(obd_class.h:1117:obd_statfs()) lustre-OST0001-osc-ffff94d405cc3800: update ffff94d407dbb7d0 cache blocks 901369/958279 objects 261733/262144 00000080:00200080:1.0:1788572448.730715:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730733:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730740:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730744:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730748:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730752:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730755:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730759:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730763:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730766:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730770:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730774:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730777:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730781:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730785:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730789:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730792:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730795:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730798:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730801:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730815:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730821:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730825:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730828:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730832:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730836:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730840:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730843:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730846:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730850:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730854:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730857:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730861:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730865:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730869:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730872:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730876:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730879:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730883:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730887:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730891:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730895:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730899:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730902:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730907:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730910:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730914:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730918:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730922:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730925:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730929:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730932:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730935:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730939:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730942:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730946:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730949:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730953:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730957:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730961:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730964:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730968:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730972:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730975:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730979:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730984:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.730987:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.730991:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.731793:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.731871:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.731880:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.731884:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.731888:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.731891:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.731895:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.731899:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.731902:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.731906:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.731910:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.731913:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.731917:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.731921:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.731924:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.731927:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.731931:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.731934:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.731938:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.731941:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.731946:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.731949:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.731953:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.731957:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.731960:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.731964:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.731967:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.731970:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.731974:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.731978:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.731981:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.731985:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.731989:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.731992:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.732180:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.732188:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.732194:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.732197:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.732223:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.732228:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.732233:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.732236:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.732241:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.732245:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.732248:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.732251:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.732254:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.732258:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.732261:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.732264:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.732268:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.732270:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.732274:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.732277:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.732281:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.732284:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200080:1.0:1788572448.732287:0:67208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc916ac2f0 00020000:00000080:1.0:1788572448.732291:0:67208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d57c00 uarg=0000000000000000 00000080:00200008:1.0:1788572448.732305:0:67208:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 100000 00000080:00200000:1.0:1788572448.803671:0:67224:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572448.804165:0:67224:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572448.804187:0:67224:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572448.804235:0:67224:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1788572448.804244:0:67224:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572448.804254:0:67224:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572448.804257:0:67224:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0a4a 00000080:00010000:1.0:1788572448.804265:0:67224:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a4a from it ffffb68a4106fbf8 00010000:00010000:1.0:1788572448.804269:0:67224:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572448.804283:0:67224:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1788572448.804299:0:67224:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572448.804405:0:67224:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d130b.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), flags=27 00000080:00200000:1.0:1788572448.804411:0:67224:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d130b.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), intent=0 00000002:00010000:1.0:1788572448.804435:0:67224:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d130b.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1788572448.804548:0:67224:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a51 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 67224 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572448.804556:0:67224:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a51 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 67224 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572448.804568:0:67224:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a51 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 67224 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572448.804598:0:67224:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67224:1875450488499200:192.168.202.125@tcp:101:lfs.0 00000100:00100000:1.0:1788572448.804748:0:67224:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4110f9900 going to sleep for 11 seconds 00000100:00100000:1.0:1788572448.818673:0:67224:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67224:1875450488499200:192.168.202.125@tcp:101:lfs.0 00010000:00010000:1.0:1788572448.818711:0:67224:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a51 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0340 expref: -99 pid: 67224 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572448.818734:0:67224:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a51 lrc: 4/1,0 mode: --/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0340 expref: -99 pid: 67224 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572448.818742:0:67224:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a51 lrc: 4/1,0 mode: --/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0340 expref: -99 pid: 67224 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572448.818755:0:67224:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a51 lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0340 expref: -99 pid: 67224 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572448.818766:0:67224:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a51 lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0340 expref: -99 pid: 67224 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572448.818782:0:67224:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a51 lrc: 5/2,0 mode: PR/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0340 expref: -99 pid: 67224 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572448.818789:0:67224:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a51 lrc: 5/2,0 mode: PR/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0340 expref: -99 pid: 67224 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572448.818796:0:67224:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a51 lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0340 expref: -99 pid: 67224 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572448.818937:0:67224:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff94d4049be300 x1875450488499200/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 592/688 e 0 to 0 dl 1788572464 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572448.818954:0:67224:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a51 lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0340 expref: -99 pid: 67224 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572448.818966:0:67224:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589977617/3 (0 0) 00000080:00200000:1.0:1788572448.818975:0:67224:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572448.819024:0:67224:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x20000a811:0x3:0x0], updating i_size 4096 00000080:00200000:1.0:1788572448.819028:0:67224:0:(namei.c:147:ll_iget()) got inode: [0x20000a811:0x3:0x0](ffff94d4034300c8): rc = 0 00000080:00010000:1.0:1788572448.819035:0:67224:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a811:0x3:0x0](ffff94d4034300c8) for lock 0x1e260726fbbc0a51 00000080:00010000:1.0:1788572448.819050:0:67224:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a811:0x3:0x0](ffff94d4034300c8) 00000080:00010000:1.0:1788572448.819054:0:67224:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a51 from it ffffb68a4106fac8 00010000:00010000:1.0:1788572448.819057:0:67224:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a51 lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0340 expref: -99 pid: 67224 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572448.819066:0:67224:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a51 lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0340 expref: -99 pid: 67224 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572448.819096:0:67224:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x20000a811:0x3:0x0](ffff94d4034300c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788572448.819115:0:67224:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f130b.replay-single, dir=[0x20000a811:0x3:0x0](ffff94d4034300c8), file ffff94d414fba480, open_flags 18800, mode 0 opened 0 00000080:00200000:1.0:1788572448.819121:0:67224:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f130b.replay-single, dir=[0x20000a811:0x3:0x0](ffff94d4034300c8), intent=open 00000002:00010000:1.0:1788572448.819141:0:67224:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f130b.replay-single,[0x0:0x0:0x0]) in obj [0x20000a811:0x3:0x0], intent: open flags 0304001 00010000:00010000:1.0:1788572448.819192:0:67224:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582200/0x1e260726fbbc0a58 lrc: 3/1,0 mode: --/CR res: [0x20000a811:0x3:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 67224 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572448.819234:0:67224:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582200/0x1e260726fbbc0a58 lrc: 3/1,0 mode: --/CR res: [0x20000a811:0x3:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 67224 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572448.819247:0:67224:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572448.819255:0:67224:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d4049be300 x1875450488499456/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 752/131944 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572448.819263:0:67224:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582200/0x1e260726fbbc0a58 lrc: 3/1,0 mode: --/CR res: [0x20000a811:0x3:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 67224 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572448.819284:0:67224:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67224:1875450488499456:192.168.202.125@tcp:101:lfs.0 00000100:00100000:1.0:1788572448.819870:0:67224:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4110f9900 going to sleep for 11 seconds 00000100:00100000:3.0:1788572448.827867:0:67224:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67224:1875450488499456:192.168.202.125@tcp:101:lfs.0 00010000:00010000:3.0:1788572448.827891:0:67224:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582200/0x1e260726fbbc0a58 lrc: 4/1,0 mode: --/CR res: [0x20000a811:0x3:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 67224 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572448.827905:0:67224:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582200/0x1e260726fbbc0a58 lrc: 4/1,0 mode: --/CR res: [0x20000a811:0x3:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 67224 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572448.827912:0:67224:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582200/0x1e260726fbbc0a58 lrc: 4/1,0 mode: --/CR res: [0x20000a811:0x3:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 67224 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572448.827920:0:67224:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582200/0x1e260726fbbc0a58 lrc: 3/0,0 mode: --/CR res: [0x20000a811:0x3:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 67224 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572448.827929:0:67224:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582200/0x1e260726fbbc0a58 lrc: 4/0,0 mode: --/CR res: [0x20000a811:0x3:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 67224 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572448.827939:0:67224:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412582200 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:3.0:1788572448.827943:0:67224:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582200/0x1e260726fbbc0a58 lrc: 4/0,0 mode: --/CR res: [0x20000a811:0x3:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 67224 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572448.827950:0:67224:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582200/0x1e260726fbbc0a58 lrc: 4/0,0 mode: --/CR res: [0x20000a811:0x3:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 67224 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572448.827957:0:67224:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582200/0x1e260726fbbc0a58 lrc: 5/0,0 mode: --/CR res: [0x20000a811:0x3:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 67224 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572448.827965:0:67224:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572448.827972:0:67224:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582200/0x1e260726fbbc0a58 lrc: 3/0,0 mode: --/CR res: [0x20000a811:0x3:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 67224 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572448.827978:0:67224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582200/0x1e260726fbbc0a58 lrc: 0/0,0 mode: --/CR res: [0x20000a811:0x3:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 67224 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572448.828006:0:67224:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0xb, status=-20 req@ffff94d4049be300 x1875450488499456/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 752/608 e 0 to 0 dl 1788572464 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788572448.828019:0:67224:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000a811:0x3:0x0] suppgids 0 -1: rc -20 00000080:00200000:3.0:1788572448.828168:0:67224:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:3.0:1788572448.828187:0:67224:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572448.828227:0:67224:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1788572448.828238:0:67224:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:3.0:1788572448.828245:0:67224:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572448.828251:0:67224:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:3.0:1788572448.828254:0:67224:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0a4a 00000080:00010000:3.0:1788572448.828259:0:67224:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a4a from it ffffb68a4106fbf8 00010000:00010000:3.0:1788572448.828263:0:67224:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1788572448.828270:0:67224:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:3.0:1788572448.828292:0:67224:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:3.0:1788572448.828300:0:67224:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d130b.replay-single, flags=267 00000080:00200000:3.0:1788572448.828306:0:67224:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x20000a811:0x3:0x0](ffff94d4034300c8), inode mode 41ed mask 44 00000080:00200008:3.0:1788572448.828609:0:67224:0:(file.c:1090:ll_file_open()) START file d130b.replay-single:[0x20000a811:0x3:0x0](ffff94d4034300c8), flags 304000 00000002:00010000:3.0:1788572448.828629:0:67224:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x20000a811:0x3:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240304001 00010000:00010000:3.0:1788572448.828674:0:67224:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030000/0x1e260726fbbc0a5f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 67224 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572448.828682:0:67224:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030000/0x1e260726fbbc0a5f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 67224 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788572448.828692:0:67224:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788572448.828765:0:67224:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c57100 x1875450488499712/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 736/131944 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788572448.828776:0:67224:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030000/0x1e260726fbbc0a5f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 67224 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572448.828797:0:67224:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67224:1875450488499712:192.168.202.125@tcp:101:lfs.0 00000100:00100000:3.0:1788572448.829937:0:67224:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d412842180 going to sleep for 11 seconds 00000100:00100000:0.0:1788572448.845971:0:67224:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67224:1875450488499712:192.168.202.125@tcp:101:lfs.0 00010000:00010000:0.0:1788572448.848740:0:67224:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030000/0x1e260726fbbc0a5f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 67224 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572448.848762:0:67224:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030000/0x1e260726fbbc0a5f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 67224 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572448.848771:0:67224:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030000/0x1e260726fbbc0a5f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 67224 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572448.848779:0:67224:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030000/0x1e260726fbbc0a5f lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 67224 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572448.848787:0:67224:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030000/0x1e260726fbbc0a5f lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 67224 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572448.848793:0:67224:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d400030000 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:0.0:1788572448.848798:0:67224:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030000/0x1e260726fbbc0a5f lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 67224 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572448.848805:0:67224:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030000/0x1e260726fbbc0a5f lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 67224 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572448.848814:0:67224:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030000/0x1e260726fbbc0a5f lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 67224 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572448.848821:0:67224:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788572448.848827:0:67224:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030000/0x1e260726fbbc0a5f lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 67224 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572448.848833:0:67224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030000/0x1e260726fbbc0a5f lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 67224 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572448.848858:0:67224:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff94d408c57100 x1875450488499712/t103079215105(103079215105) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 736/664 e 0 to 0 dl 1788572464 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788572448.848872:0:67224:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d408c57100 x1875450488499712/t103079215105(103079215105) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 736/664 e 0 to 0 dl 1788572464 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572448.848889:0:67224:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788572448.848917:0:67224:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x20000a811:0x3:0x0], updating i_size 4096 00000002:00100000:0.0:1788572448.848936:0:67224:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d408c57100 x1875450488499712/t103079215105(103079215105) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572464 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:0.0:1788572448.849039:0:67224:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x20000a811:0x3:0x0](ffff94d4034300c8) cmd=c0086915 arg=14a43d0 00000100:00100000:0.0:1788572448.849103:0:67224:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67224:1875450488499840:192.168.202.125@tcp:34:lfs.0 00000100:00100000:0.0:1788572448.849183:0:67224:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d407e13180 going to sleep for 11 seconds 00000100:00100000:0.0:1788572448.855949:0:67224:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67224:1875450488499840:192.168.202.125@tcp:34:lfs.0 00000080:00200008:0.0:1788572448.856136:0:67224:0:(file.c:479:ll_file_release()) START file d130b.replay-single:[0x20000a811:0x3:0x0](ffff94d4034300c8), flags 304000 00000002:00100000:0.0:1788572448.856225:0:67224:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff94d408c57100 x1875450488499712/t103079215105(103079215105) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572464 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788572448.856237:0:67224:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572448.856243:0:67224:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408d3c700 x1875450488500096/t0(0) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/712 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788572448.856265:0:67224:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67224:1875450488500096:192.168.202.125@tcp:35:lfs.0 00000100:00100000:0.0:1788572448.856349:0:67224:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d407e13180 going to sleep for 11 seconds 00000100:00100000:0.0:1788572448.870984:0:67224:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67224:1875450488500096:192.168.202.125@tcp:35:lfs.0 00000002:00100000:0.0:1788572448.871022:0:67224:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff94d408c57100 x1875450488499712/t103079215105(103079215105) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572464 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788572448.971702:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488500224:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572448.976321:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488500224:192.168.202.125@tcp:400:kworker.0 00010000:00010000:2.0:1788572449.416327:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572450.437387:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572451.461336:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572452.485810:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572453.512308:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572454.085729:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488500480:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572454.085831:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488500736:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572454.085867:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:3.0:1788572454.085873:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488500352:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572454.085904:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488500608:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572454.085925:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488500864:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572454.089954:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488500352:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572454.089980:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488500608:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572454.089992:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488500864:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572454.090003:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488500480:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572454.090014:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488500736:192.168.202.125@tcp:400:kworker.0 00010000:00010000:3.0:1788572454.536289:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572455.558129:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572456.581729:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1788572456.853155:0:67798:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 131a: DoM file write lock replay === 21:40:55 (1788572455) 00000080:00200000:1.0:1788572457.074398:0:67812:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572457.074432:0:67812:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572457.074452:0:67812:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1788572457.074481:0:67812:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:1.0:1788572457.074490:0:67812:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572457.074567:0:67812:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572457.074572:0:67812:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0a4a 00000080:00010000:1.0:1788572457.074581:0:67812:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a4a from it ffffb68a42077b58 00010000:00010000:1.0:1788572457.074586:0:67812:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1788572457.074597:0:67812:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:1.0:1788572457.074615:0:67812:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572457.074631:0:67812:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f131a.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), flags=8 00000080:00200000:1.0:1788572457.074635:0:67812:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f131a.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), intent=getattr 00000002:00010000:1.0:1788572457.074647:0:67812:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f131a.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788572457.075188:0:67812:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582000/0x1e260726fbbc0a66 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 67812 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572457.075199:0:67812:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582000/0x1e260726fbbc0a66 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 67812 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572457.075229:0:67812:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582000/0x1e260726fbbc0a66 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 67812 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572457.075256:0:67812:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67812:1875450488500992:192.168.202.125@tcp:101:lfs.0 00000100:00100000:1.0:1788572457.075346:0:67812:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408d81840 going to sleep for 11 seconds 00000100:00100000:1.0:1788572457.087928:0:67812:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 103079215106 gen 4 00000100:00100000:1.0:1788572457.087945:0:67812:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67812:1875450488500992:192.168.202.125@tcp:101:lfs.0 00010000:00010000:1.0:1788572457.087965:0:67812:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582000/0x1e260726fbbc0a66 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 67812 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572457.087979:0:67812:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582000/0x1e260726fbbc0a66 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 67812 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572457.087986:0:67812:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582000/0x1e260726fbbc0a66 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 67812 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572457.087993:0:67812:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582000/0x1e260726fbbc0a66 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 67812 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572457.088002:0:67812:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582000/0x1e260726fbbc0a66 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 67812 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572457.088009:0:67812:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412582000 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:1.0:1788572457.088014:0:67812:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582000/0x1e260726fbbc0a66 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 67812 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572457.088069:0:67812:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582000/0x1e260726fbbc0a66 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 67812 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572457.088144:0:67812:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582000/0x1e260726fbbc0a66 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 67812 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572457.088152:0:67812:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572457.088159:0:67812:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582000/0x1e260726fbbc0a66 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 67812 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572457.088166:0:67812:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582000/0x1e260726fbbc0a66 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 67812 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572457.088182:0:67812:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff94d4049bdc00 x1875450488500992/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 592/584 e 0 to 0 dl 1788572473 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572457.088197:0:67812:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1788572457.088238:0:67812:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:1.0:1788572457.088252:0:67812:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00010000:1.0:1788572457.088261:0:67812:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a4a from it ffffb68a42077930 00010000:00010000:1.0:1788572457.088265:0:67812:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:1.0:1788572457.088273:0:67812:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:1.0:1788572457.088342:0:67812:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572457.088355:0:67812:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572457.088362:0:67812:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:1.0:1788572457.088369:0:67812:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:1.0:1788572457.088379:0:67812:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572457.088383:0:67812:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572457.088386:0:67812:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0a4a 00000080:00010000:1.0:1788572457.088390:0:67812:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a4a from it ffffb68a42077bf8 00010000:00010000:1.0:1788572457.088393:0:67812:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:1.0:1788572457.088399:0:67812:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:1.0:1788572457.088406:0:67812:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572457.088415:0:67812:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f131a.replay-single, flags=777 00000080:00200000:1.0:1788572457.088419:0:67812:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572457.088424:0:67812:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572457.088430:0:67812:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1788572457.088437:0:67812:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:1.0:1788572457.088442:0:67812:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572457.088445:0:67812:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572457.088448:0:67812:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0a4a 00000080:00010000:1.0:1788572457.088450:0:67812:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a4a from it ffffb68a42077b38 00010000:00010000:1.0:1788572457.088453:0:67812:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1788572457.088475:0:67812:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:1.0:1788572457.088482:0:67812:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 3 00000080:00200000:1.0:1788572457.088486:0:67812:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f131a.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), file ffff94d402be9980, open_flags a141, mode 81b6 opened 0 00000080:00200000:1.0:1788572457.088494:0:67812:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f131a.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), intent=open|creat 00000002:00010000:1.0:1788572457.089236:0:67812:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f131a.replay-single,[0x20000a811:0x6:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0120502 00010000:00010000:1.0:1788572457.089253:0:67812:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:1.0:1788572457.089307:0:67812:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:1.0:1788572457.089315:0:67812:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572457.089318:0:67812:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031000/0x1e260726fbbc0a4a lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4e0316 expref: -99 pid: 66741 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:1.0:1788572457.089350:0:67812:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc0a6d lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 67812 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572457.089354:0:67812:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc0a6d lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 67812 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572457.089363:0:67812:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572457.089371:0:67812:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d4049bdc00 x1875450488501248/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 536/66408 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572457.089377:0:67812:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc0a6d lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 67812 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572457.089396:0:67812:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67812:1875450488501248:192.168.202.125@tcp:101:lfs.0 00000100:00100000:1.0:1788572457.089801:0:67812:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408d81840 going to sleep for 11 seconds 00000100:00100000:1.0:1788572457.096732:0:67812:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67812:1875450488501248:192.168.202.125@tcp:101:lfs.0 00010000:00010000:1.0:1788572457.096760:0:67812:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc0a6d lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03a9 expref: -99 pid: 67812 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572457.096788:0:67812:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc0a6d lrc: 4/0,1 mode: --/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03a9 expref: -99 pid: 67812 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572457.096796:0:67812:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc0a6d lrc: 4/0,1 mode: --/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03a9 expref: -99 pid: 67812 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572457.096816:0:67812:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc0a6d lrc: 4/0,1 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e03a9 expref: -99 pid: 67812 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572457.096828:0:67812:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc0a6d lrc: 4/0,1 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03a9 expref: -99 pid: 67812 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572457.096843:0:67812:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc0a6d lrc: 5/1,1 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e03a9 expref: -99 pid: 67812 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572457.096852:0:67812:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc0a6d lrc: 5/1,1 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e03a9 expref: -99 pid: 67812 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572457.096859:0:67812:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc0a6d lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e03a9 expref: -99 pid: 67812 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572457.096875:0:67812:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x8000037, status=0 req@ffff94d4049bdc00 x1875450488501248/t103079215107(103079215107) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 536/608 e 0 to 0 dl 1788572473 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572457.096888:0:67812:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d4049bdc00 x1875450488501248/t103079215107(103079215107) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 536/608 e 0 to 0 dl 1788572473 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572457.096901:0:67812:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc0a6d lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03a9 expref: -99 pid: 67812 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572457.096911:0:67812:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589977617/6 (0 0) 00000080:00200000:1.0:1788572457.096917:0:67812:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572457.096929:0:67812:0:(namei.c:1628:ll_create_it()) VFS Op:name=f131a.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), intent=open|creat 00000080:00200000:1.0:1788572457.096959:0:67812:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x20000a811:0x6:0x0](ffff94d4034963c8) 00000080:00200000:1.0:1788572457.096966:0:67812:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x20000a811:0x6:0x0], updating i_size 0 00000080:00200000:1.0:1788572457.097051:0:67812:0:(file.c:6826:ll_layout_conf()) [0x20000a811:0x6:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:1.0:1788572457.097062:0:67812:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x20000a811:0x6:0x0](ffff94d4034963c8) 00000080:00010000:1.0:1788572457.097066:0:67812:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a811:0x6:0x0](ffff94d4034963c8) for lock 0x1e260726fbbc0a6d 00000080:00010000:1.0:1788572457.097075:0:67812:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a811:0x6:0x0](ffff94d4034963c8) 00000080:00200008:1.0:1788572457.097086:0:67812:0:(file.c:1090:ll_file_open()) START file f131a.replay-single:[0x20000a811:0x6:0x0](ffff94d4034963c8), flags 120501 00000002:00100000:1.0:1788572457.097105:0:67812:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d4049bdc00 x1875450488501248/t103079215107(103079215107) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 536/608 e 0 to 0 dl 1788572473 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788572457.097146:0:67812:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a6d from it ffff94d408d4a360 00010000:00010000:1.0:1788572457.097151:0:67812:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc0a6d lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4e03a9 expref: -99 pid: 67812 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572457.097160:0:67812:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc0a6d lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e03a9 expref: -99 pid: 67812 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572457.097184:0:67812:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x20000a811:0x6:0x0](ffff94d4034963c8), inode mode 81a4 mask 40 00000080:00200000:1.0:1788572457.097492:0:67812:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x20000a811:0x6:0x0](ffff94d4034963c8), inode mode 81a4 mask 2 00000080:00200000:1.0:1788572457.097503:0:67812:0:(xattr.c:413:ll_xattr_set()) VFS Op:inode=[0x20000a811:0x6:0x0](ffff94d4034963c8), xattr lov 00000002:00010000:1.0:1788572457.097519:0:67812:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x20000a811:0x6:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 040000002 00010000:00010000:1.0:1788572457.097565:0:67812:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582000/0x1e260726fbbc0a74 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 67812 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572457.097574:0:67812:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582000/0x1e260726fbbc0a74 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 67812 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572457.097585:0:67812:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572457.097592:0:67812:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d4049bd880 x1875450488501504/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 736/131944 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572457.097600:0:67812:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582000/0x1e260726fbbc0a74 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 67812 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572457.097620:0:67812:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67812:1875450488501504:192.168.202.125@tcp:101:lfs.0 00000100:00100000:1.0:1788572457.097784:0:67812:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408d81cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572457.101924:0:2397:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.125@tcp (source: 12345-192.168.202.125@tcp) 00000100:00100000:2.0:1788572457.102984:0:42439:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875450511477120 00000100:00100000:2.0:1788572457.102998:0:42439:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875450511477120 00000100:00100000:2.0:1788572457.103068:0:42439:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.125@tcp, seq: 91 00000100:00100000:2.0:1788572457.103088:0:42439:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff94d408c56d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:9bf287c0-f582-4b83-84c1-15e766218620+5:80444:x1875450511477120:12345-192.168.202.125@tcp:104: 00010000:00010000:2.0:1788572457.103111:0:42439:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc0a6d lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xec901abfeb4e03a9 expref: -99 pid: 67812 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572457.103229:0:42439:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788572457.103237:0:42439:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff94d408c56d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:9bf287c0-f582-4b83-84c1-15e766218620+5:80444:x1875450511477120:12345-192.168.202.125@tcp:104: Request processed in 160us (1324us total) trans 0 rc 0/0 00000100:00100000:2.0:1788572457.103245:0:42439:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.125@tcp, seq: 91 00010000:00010000:0.0:1788572457.103307:0:42449:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788572457.103315:0:42449:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc0a6d lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xec901abfeb4e03a9 expref: -99 pid: 67812 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572457.103324:0:42449:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412582400 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:0.0:1788572457.103328:0:42449:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc0a6d lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xec901abfeb4e03a9 expref: -99 pid: 67812 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572457.103337:0:42449:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc0a6d lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xec901abfeb4e03a9 expref: -99 pid: 67812 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572457.103345:0:42449:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc0a6d lrc: 3/0,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0xec901abfeb4e03a9 expref: -99 pid: 67812 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572457.103394:0:42449:0:(vvp_object.c:117:vvp_conf_set()) [0x20000a811:0x6:0x0]: losing layout lock 00010000:00010000:0.0:1788572457.103454:0:42449:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc0a6d lrc: 2/0,0 mode: --/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xec901abfeb4e03a9 expref: -99 pid: 67812 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572457.103476:0:42449:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572457.103502:0:42449:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc0a6d lrc: 1/0,0 mode: --/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xec901abfeb4e03a9 expref: -99 pid: 67812 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572457.103510:0:42449:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582400/0x1e260726fbbc0a6d lrc: 0/0,0 mode: --/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xec901abfeb4e03a9 expref: -99 pid: 67812 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572457.103526:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572457.103528:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572457.103529:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572457.103537:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1788572457.103551:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488501632:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572457.108360:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488501632:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572457.113366:0:67812:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67812:1875450488501504:192.168.202.125@tcp:101:lfs.0 00010000:00010000:1.0:1788572457.113393:0:67812:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582000/0x1e260726fbbc0a74 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 67812 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572457.113408:0:67812:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582000/0x1e260726fbbc0a74 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 67812 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572457.113415:0:67812:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582000/0x1e260726fbbc0a74 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 67812 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572457.113427:0:67812:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582000/0x1e260726fbbc0a74 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 67812 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572457.113436:0:67812:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582000/0x1e260726fbbc0a74 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 67812 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572457.113443:0:67812:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412582000 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:1.0:1788572457.113447:0:67812:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582000/0x1e260726fbbc0a74 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 67812 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572457.113453:0:67812:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582000/0x1e260726fbbc0a74 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 67812 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572457.113469:0:67812:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582000/0x1e260726fbbc0a74 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 67812 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572457.113477:0:67812:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572457.113484:0:67812:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582000/0x1e260726fbbc0a74 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 67812 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572457.113491:0:67812:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582000/0x1e260726fbbc0a74 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 67812 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572457.113518:0:67812:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x800002b, status=0 req@ffff94d4049bd880 x1875450488501504/t103079215108(103079215108) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 736/824 e 0 to 0 dl 1788572473 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572457.113532:0:67812:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d4049bd880 x1875450488501504/t103079215108(103079215108) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 736/824 e 0 to 0 dl 1788572473 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572457.113549:0:67812:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572457.113640:0:67812:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788572457.113649:0:67812:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572457.113736:0:67812:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x20000a811:0x6:0x0], updating i_size 0 00000002:00100000:1.0:1788572457.113754:0:67812:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d4049bd880 x1875450488501504/t103079215108(103079215108) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 736/824 e 0 to 0 dl 1788572473 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572457.113789:0:67812:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff94d4049bd880 x1875450488501504/t103079215108(103079215108) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 736/824 e 0 to 0 dl 1788572473 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572457.113798:0:67812:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572457.113804:0:67812:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d4049bc700 x1875450488501888/t0(0) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/712 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572457.113825:0:67812:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67812:1875450488501888:192.168.202.125@tcp:35:lfs.0 00000100:00100000:1.0:1788572457.113900:0:67812:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408d81cc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572457.128415:0:67812:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67812:1875450488501888:192.168.202.125@tcp:35:lfs.0 00000002:00100000:1.0:1788572457.128453:0:67812:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff94d4049bd880 x1875450488501504/t103079215108(103079215108) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 736/824 e 0 to 0 dl 1788572473 ref 2 fl Complete:RQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1788572457.128519:0:67812:0:(file.c:479:ll_file_release()) START file f131a.replay-single:[0x20000a811:0x6:0x0](ffff94d4034963c8), flags 100001 00000002:00100000:1.0:1788572457.128569:0:67812:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff94d4049bdc00 x1875450488501248/t103079215107(103079215107) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 536/608 e 0 to 0 dl 1788572473 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572457.128579:0:67812:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572457.128585:0:67812:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d4049be680 x1875450488502144/t0(0) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/712 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572457.128607:0:67812:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67812:1875450488502144:192.168.202.125@tcp:35:lfs.0 00000100:00100000:1.0:1788572457.128686:0:67812:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408d81840 going to sleep for 11 seconds 00000100:00100000:1.0:1788572457.134580:0:67812:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67812:1875450488502144:192.168.202.125@tcp:35:lfs.0 00000002:00100000:1.0:1788572457.134624:0:67812:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff94d4049bdc00 x1875450488501248/t103079215107(103079215107) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 536/608 e 0 to 0 dl 1788572473 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1788572457.605411:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:3.0:1788572458.212835:0:67823:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572458.212850:0:67823:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:3.0:1788572458.212874:0:67823:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572458.212939:0:67823:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 67823 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.212950:0:67823:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 67823 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.212960:0:67823:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 67823 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572458.212987:0:67823:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67823:1875450488502272:192.168.202.125@tcp:101:lfs.0 00000100:00100000:3.0:1788572458.213088:0:67823:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408501180 going to sleep for 11 seconds 00000100:00100000:3.0:1788572458.224139:0:67823:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 103079215110 gen 4 00000100:00100000:3.0:1788572458.224255:0:67823:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67823:1875450488502272:192.168.202.125@tcp:101:lfs.0 00010000:00010000:3.0:1788572458.224274:0:67823:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.224287:0:67823:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.224300:0:67823:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.224310:0:67823:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.224322:0:67823:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.224330:0:67823:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.224344:0:67823:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572458.224354:0:67823:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408c56300 x1875450488502272/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572474 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572458.224368:0:67823:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.224380:0:67823:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788572458.224388:0:67823:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572458.224414:0:67823:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:3.0:1788572458.224420:0:67823:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0a7b 00000080:00010000:3.0:1788572458.224426:0:67823:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:3.0:1788572458.224429:0:67823:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a7b from it ffffb68a4170bcb8 00010000:00010000:3.0:1788572458.224432:0:67823:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.224442:0:67823:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788572458.225138:0:67823:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788572458.225145:0:67823:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:3.0:1788572458.225162:0:67823:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572458.225172:0:67823:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572458.225189:0:67823:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572458.225196:0:67823:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572458.225227:0:67823:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:3.0:1788572458.225231:0:67823:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0a7b 00000080:00010000:3.0:1788572458.225236:0:67823:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a7b from it ffffb68a4170bcb8 00010000:00010000:3.0:1788572458.225240:0:67823:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572458.225247:0:67823:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572458.225363:0:67823:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:3.0:1788572458.225372:0:67823:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572458.225379:0:67823:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572458.225388:0:67823:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572458.225394:0:67823:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572458.225398:0:67823:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:3.0:1788572458.225401:0:67823:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0a7b 00000080:00010000:3.0:1788572458.225405:0:67823:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a7b from it ffffb68a4170bbb0 00010000:00010000:3.0:1788572458.225408:0:67823:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572458.225417:0:67823:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788572458.225426:0:67823:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 44 00000080:00200008:3.0:1788572458.225433:0:67823:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 100000 00000080:00200080:3.0:1788572458.225549:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:3.0:1788572458.225564:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f31400 uarg=0000000000000000 00800000:00000004:3.0:1788572458.225572:0:67823:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0000-mdc-ffff94d405cc3800: age 3602, max_age 3610 00000100:00100000:3.0:1788572458.225614:0:67823:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67823:1875450488502400:192.168.202.125@tcp:41:lfs.0 00000100:00100000:3.0:1788572458.225712:0:67823:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408501180 going to sleep for 11 seconds 00000100:00100000:3.0:1788572458.229729:0:67823:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67823:1875450488502400:192.168.202.125@tcp:41:lfs.0 00800000:00000004:3.0:1788572458.229759:0:67823:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0000-mdc-ffff94d405cc3800: update ffff94d412909e10 cache blocks 317131/353529 objects 1023603/1024000 00000080:00200080:3.0:1788572458.229839:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:3.0:1788572458.229856:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d416f31400 uarg=0000000000000000 00800000:00000004:3.0:1788572458.229865:0:67823:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0001-mdc-ffff94d405cc3800: age 3602, max_age 3610 00000100:00100000:3.0:1788572458.229902:0:67823:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67823:1875450488502528:192.168.202.125@tcp:41:lfs.0 00000100:00100000:3.0:1788572458.229992:0:67823:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408501180 going to sleep for 11 seconds 00000100:00100000:0.0:1788572458.236630:0:67823:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67823:1875450488502528:192.168.202.125@tcp:41:lfs.0 00800000:00000004:0.0:1788572458.236660:0:67823:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0001-mdc-ffff94d405cc3800: update ffff94d407db9e10 cache blocks 317207/353529 objects 1023684/1024000 00000080:00200080:0.0:1788572458.236737:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.236757:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.236765:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.236770:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.236774:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.236778:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.236784:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.236787:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.236791:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.236794:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.236798:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.236802:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.236806:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.236809:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.236813:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.236816:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.236820:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.236823:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.236826:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.236830:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.236836:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.236840:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.236844:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.236848:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.236852:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.236856:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.236859:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.236862:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.236867:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.236871:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.236875:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.236879:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.236883:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.236887:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.236891:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.236895:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.236899:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.236902:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.236907:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.236911:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.236915:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.236918:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.236922:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.236926:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.236930:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.236935:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.236939:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.236943:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.236947:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.236952:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.236956:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.236960:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.236964:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.236968:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.236972:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.236976:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.236980:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.236983:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.236987:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.236990:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.236994:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.236997:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237000:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.237003:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237006:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.237009:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237012:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.237015:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237019:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.237022:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237026:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.237029:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237032:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.237036:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237041:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.237044:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237048:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.237051:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237056:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.237059:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237062:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.237066:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237069:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.237072:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237076:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.237079:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237082:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.237099:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237104:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.237108:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237112:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.237115:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237119:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.237123:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237127:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.237131:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237134:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.237138:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237141:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.237146:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237149:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.237152:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237156:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.237159:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237163:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.237166:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237169:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.237172:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237175:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.237178:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237182:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.237185:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237188:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.237192:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237196:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.237200:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237222:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.237226:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237229:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.237233:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237236:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.237239:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237243:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.237246:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237249:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00800000:00000080:0.0:1788572458.237251:0:67823:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.237255:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.237261:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00020000:00000004:0.0:1788572458.237269:0:67823:0:(obd_class.h:1091:obd_statfs()) lustre-OST0000-osc-ffff94d405cc3800: age 3602, max_age 3610 00000100:00100000:0.0:1788572458.237309:0:67823:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67823:1875450488502656:192.168.202.125@tcp:13:lfs.0 00000100:00100000:0.0:1788572458.237480:0:67823:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405411a80 going to sleep for 11 seconds 00000100:00100000:0.0:1788572458.246347:0:67823:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67823:1875450488502656:192.168.202.125@tcp:13:lfs.0 00020000:00000004:0.0:1788572458.246374:0:67823:0:(obd_class.h:1117:obd_statfs()) lustre-OST0000-osc-ffff94d405cc3800: update ffff94d407dbd190 cache blocks 901369/958279 objects 261733/262144 00000080:00200080:0.0:1788572458.246494:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.246518:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00020000:00000004:0.0:1788572458.246527:0:67823:0:(obd_class.h:1091:obd_statfs()) lustre-OST0001-osc-ffff94d405cc3800: age 3602, max_age 3610 00000100:00100000:0.0:1788572458.246565:0:67823:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67823:1875450488502784:192.168.202.125@tcp:13:lfs.0 00000100:00100000:0.0:1788572458.246651:0:67823:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405411a80 going to sleep for 11 seconds 00000100:00100000:0.0:1788572458.259183:0:67823:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:67823:1875450488502784:192.168.202.125@tcp:13:lfs.0 00020000:00000004:0.0:1788572458.259261:0:67823:0:(obd_class.h:1117:obd_statfs()) lustre-OST0001-osc-ffff94d405cc3800: update ffff94d407dbb7d0 cache blocks 901369/958279 objects 261733/262144 00000080:00200080:0.0:1788572458.259322:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259338:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259346:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259350:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259355:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259358:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259362:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259366:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259370:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259374:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259378:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259382:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259386:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259401:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259406:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259410:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259413:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259417:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259421:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259424:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259428:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259431:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259435:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259439:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259443:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259447:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259451:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259455:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259458:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259463:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259476:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259481:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259485:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259488:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259492:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259496:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259500:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259503:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259507:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259510:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259514:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259518:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259522:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259525:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259530:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259533:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259537:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259541:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259544:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259548:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259552:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259555:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259559:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259562:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259566:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259570:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259573:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259577:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259581:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259584:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259588:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259592:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259596:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259599:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259604:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259607:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259611:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259615:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259619:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259622:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259625:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259629:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259633:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259636:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259641:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259644:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259649:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259652:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259656:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259660:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259664:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259668:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259671:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259675:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259679:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259683:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259687:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259690:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259693:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259696:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259700:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259703:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259707:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259710:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259725:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259730:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259734:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259738:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259742:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259746:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259750:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259753:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259758:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259761:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259766:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259769:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259773:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259777:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259781:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259784:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259788:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259791:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259795:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259798:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259802:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259806:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259810:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259813:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259816:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259820:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259824:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259827:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259831:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259835:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200080:0.0:1788572458.259839:0:67823:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fffd64f0150 00020000:00000080:0.0:1788572458.259843:0:67823:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc800 uarg=0000000000000000 00000080:00200008:0.0:1788572458.259856:0:67823:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 100000 00000080:00200000:0.0:1788572458.358019:0:67830:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:0.0:1788572458.358057:0:67830:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572458.358079:0:67830:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572458.358096:0:67830:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788572458.358105:0:67830:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572458.358114:0:67830:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572458.358117:0:67830:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0a7b 00000080:00010000:0.0:1788572458.358126:0:67830:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a7b from it ffffb68a41db3bd0 00010000:00010000:0.0:1788572458.358130:0:67830:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572458.358139:0:67830:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788572458.358155:0:67830:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572458.358170:0:67830:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg225-client.virtnet, dir=[0x200000007:0x1:0x0](ffff94d403679988), flags=1536 00000080:00200000:0.0:1788572458.358174:0:67830:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:0.0:1788572458.358180:0:67830:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572458.358187:0:67830:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788572458.358196:0:67830:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788572458.358221:0:67830:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572458.358225:0:67830:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572458.358228:0:67830:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0a7b 00000080:00010000:0.0:1788572458.358232:0:67830:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a7b from it ffffb68a41db3cd0 00010000:00010000:0.0:1788572458.358235:0:67830:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788572458.358243:0:67830:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1788572458.358252:0:67830:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 3 00000080:00200000:0.0:1788572458.358258:0:67830:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:0.0:1788572458.358264:0:67830:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572458.358270:0:67830:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1788572458.358278:0:67830:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1788572458.358284:0:67830:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572458.358288:0:67830:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572458.358292:0:67830:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0a7b 00000080:00010000:0.0:1788572458.358295:0:67830:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a7b from it ffffb68a41db3db0 00010000:00010000:0.0:1788572458.358299:0:67830:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1788572458.358306:0:67830:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:0.0:1788572458.358313:0:67830:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 3 00000080:00200000:0.0:1788572458.358319:0:67830:0:(namei.c:2129:ll_create_nd()) VFS Op:name=fsa-oleg225-client.virtnet, dir=[0x200000007:0x1:0x0](ffff94d403679988), flags=33188, excl=1 00000080:00200000:0.0:1788572458.358324:0:67830:0:(namei.c:2084:ll_mknod()) VFS Op:name=fsa-oleg225-client.virtnet, dir=[0x200000007:0x1:0x0](ffff94d403679988) mode 100644 dev 0 00010000:00010000:0.0:1788572458.358350:0:67830:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1788572458.358401:0:67830:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1788572458.358408:0:67830:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572458.358410:0:67830:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030a00/0x1e260726fbbc0a7b lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4e03d3 expref: -99 pid: 67823 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000020:00100000:0.0:1788572458.358432:0:67830:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788572458.358445:0:67830:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408d3c700 x1875450488503040/t0(0) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 528/664 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572458.358482:0:67830:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job mcreate:9bf287c0-f582-4b83-84c1-15e766218620:67830:1875450488503040:192.168.202.125@tcp:36:mcreate.0 00000100:00100000:0.0:1788572458.358569:0:67830:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405411a80 going to sleep for 11 seconds 00000100:00100000:0.0:1788572458.364643:0:67830:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job mcreate:9bf287c0-f582-4b83-84c1-15e766218620:67830:1875450488503040:192.168.202.125@tcp:36:mcreate.0 00000080:00200000:0.0:1788572458.364695:0:67830:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x20000a811:0x7:0x0](ffff94d410bcb248) 00000080:00200000:0.0:1788572458.364808:0:67830:0:(namei.c:2137:ll_create_nd()) VFS Op:name=fsa-oleg225-client.virtnet, unhashed 1 00000080:00200000:3.0:1788572458.379874:0:67833:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:3.0:1788572458.379904:0:67833:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572458.380031:0:67833:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030800/0x1e260726fbbc0a82 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 67833 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.380043:0:67833:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030800/0x1e260726fbbc0a82 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 67833 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.380129:0:67833:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030800/0x1e260726fbbc0a82 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 67833 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572458.380160:0:67833:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:67833:1875450488503168:192.168.202.125@tcp:101:rm.0 00000100:00100000:3.0:1788572458.380585:0:67833:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408501840 going to sleep for 11 seconds 00000100:00100000:3.0:1788572458.384895:0:67833:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:67833:1875450488503168:192.168.202.125@tcp:101:rm.0 00010000:00010000:3.0:1788572458.384922:0:67833:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030800/0x1e260726fbbc0a82 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03ef expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.384935:0:67833:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030800/0x1e260726fbbc0a82 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03ef expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.384947:0:67833:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030800/0x1e260726fbbc0a82 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e03ef expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.384957:0:67833:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030800/0x1e260726fbbc0a82 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03ef expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.384969:0:67833:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030800/0x1e260726fbbc0a82 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e03ef expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.384978:0:67833:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030800/0x1e260726fbbc0a82 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e03ef expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.384985:0:67833:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030800/0x1e260726fbbc0a82 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e03ef expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572458.385003:0:67833:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408c55880 x1875450488503168/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572474 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572458.385017:0:67833:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030800/0x1e260726fbbc0a82 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03ef expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.385029:0:67833:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788572458.385037:0:67833:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572458.385126:0:67833:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:3.0:1788572458.385133:0:67833:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0a82 00000080:00010000:3.0:1788572458.385139:0:67833:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:3.0:1788572458.385142:0:67833:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a82 from it ffffb68a41de7b58 00010000:00010000:3.0:1788572458.385146:0:67833:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030800/0x1e260726fbbc0a82 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e03ef expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.385157:0:67833:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030800/0x1e260726fbbc0a82 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03ef expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572458.385774:0:67833:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:3.0:1788572458.385796:0:67833:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg225-client.virtnet, dir=[0x200000007:0x1:0x0](ffff94d403679988), flags=8 00000080:00200000:3.0:1788572458.385801:0:67833:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=fsa-oleg225-client.virtnet, dir=[0x200000007:0x1:0x0](ffff94d403679988), intent=getattr 00000002:00010000:3.0:1788572458.385823:0:67833:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: fsa-oleg225-client.virtnet,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788572458.385863:0:67833:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc0a89 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 67833 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.385872:0:67833:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc0a89 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 67833 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.385886:0:67833:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc0a89 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 67833 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572458.385906:0:67833:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:67833:1875450488503296:192.168.202.125@tcp:101:rm.0 00000100:00100000:3.0:1788572458.386013:0:67833:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408501840 going to sleep for 11 seconds 00000100:00100000:3.0:1788572458.390168:0:67833:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:67833:1875450488503296:192.168.202.125@tcp:101:rm.0 00010000:00010000:3.0:1788572458.390537:0:67833:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc0a89 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e040b expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.390561:0:67833:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc0a89 lrc: 4/1,0 mode: --/PR res: [0x20000a811:0x7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e040b expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.390569:0:67833:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc0a89 lrc: 4/1,0 mode: --/PR res: [0x20000a811:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e040b expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.390580:0:67833:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc0a89 lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e040b expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.390590:0:67833:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc0a89 lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e040b expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.390604:0:67833:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc0a89 lrc: 5/2,0 mode: PR/PR res: [0x20000a811:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e040b expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.390612:0:67833:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc0a89 lrc: 5/2,0 mode: PR/PR res: [0x20000a811:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e040b expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.390623:0:67833:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc0a89 lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e040b expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572458.390636:0:67833:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408c55880 x1875450488503296/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 600/584 e 0 to 0 dl 1788572474 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572458.390650:0:67833:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc0a89 lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e040b expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.390661:0:67833:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589977617/7 (0 0) 00000080:00200000:3.0:1788572458.390668:0:67833:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788572458.390690:0:67833:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x20000a811:0x7:0x0], updating i_size 0 00000080:00200000:3.0:1788572458.390693:0:67833:0:(namei.c:147:ll_iget()) got inode: [0x20000a811:0x7:0x0](ffff94d410bcb248): rc = 0 00000080:00200000:3.0:1788572458.390738:0:67833:0:(file.c:6826:ll_layout_conf()) [0x20000a811:0x7:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1788572458.390759:0:67833:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a811:0x7:0x0](ffff94d410bcb248) for lock 0x1e260726fbbc0a89 00000080:00010000:3.0:1788572458.390770:0:67833:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a811:0x7:0x0](ffff94d410bcb248) 00000080:00010000:3.0:1788572458.390773:0:67833:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a89 from it ffffb68a41de7b20 00010000:00010000:3.0:1788572458.390776:0:67833:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc0a89 lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4e040b expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.390786:0:67833:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc0a89 lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e040b expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788572458.390823:0:67833:0:(file.c:6285:ll_getattr_dentry()) START file fsa-oleg225-client.virtnet:[0x20000a811:0x7:0x0](ffff94d410bcb248), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572458.390831:0:67833:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x20000a811:0x7:0x0](ffff94d410bcb248),name=fsa-oleg225-client.virtnet 00000002:00010000:3.0:1788572458.390852:0:67833:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x20000a811:0x7:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572458.390862:0:67833:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc0a89 lrc: 2/1,0 mode: PR/PR res: [0x20000a811:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e040b expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572458.390874:0:67833:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc0a89 lrc: 2/1,0 mode: PR/PR res: [0x20000a811:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e040b expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572458.390881:0:67833:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572458.390887:0:67833:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a811:0x7:0x0](ffff94d410bcb248) 00000080:00010000:3.0:1788572458.390890:0:67833:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a811:0x7:0x0](ffff94d410bcb248) for lock 0x1e260726fbbc0a89 00000080:00010000:3.0:1788572458.390893:0:67833:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a89 from it ffffb68a41de7cb8 00010000:00010000:3.0:1788572458.390896:0:67833:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc0a89 lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4e040b expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572458.390903:0:67833:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc0a89 lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e040b expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572458.390959:0:67833:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:3.0:1788572458.390966:0:67833:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572458.390973:0:67833:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030800/0x1e260726fbbc0a82 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03ef expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572458.390981:0:67833:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030800/0x1e260726fbbc0a82 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03ef expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572458.390988:0:67833:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572458.390991:0:67833:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:3.0:1788572458.390993:0:67833:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0a82 00000080:00010000:3.0:1788572458.390996:0:67833:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a82 from it ffffb68a41de7c10 00010000:00010000:3.0:1788572458.390999:0:67833:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030800/0x1e260726fbbc0a82 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e03ef expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572458.391007:0:67833:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030800/0x1e260726fbbc0a82 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03ef expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572458.391015:0:67833:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:3.0:1788572458.391023:0:67833:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=fsa-oleg225-client.virtnet, flags=0 00000080:00200000:3.0:1788572458.391027:0:67833:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:3.0:1788572458.391033:0:67833:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572458.391038:0:67833:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030800/0x1e260726fbbc0a82 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03ef expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572458.391046:0:67833:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030800/0x1e260726fbbc0a82 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03ef expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572458.391066:0:67833:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572458.391070:0:67833:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:3.0:1788572458.391072:0:67833:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0a82 00000080:00010000:3.0:1788572458.391075:0:67833:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a82 from it ffffb68a41de7d68 00010000:00010000:3.0:1788572458.391078:0:67833:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030800/0x1e260726fbbc0a82 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e03ef expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572458.391085:0:67833:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030800/0x1e260726fbbc0a82 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e03ef expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788572458.391092:0:67833:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 3 00000080:00200000:3.0:1788572458.391098:0:67833:0:(namei.c:2418:ll_unlink()) VFS Op:name=fsa-oleg225-client.virtnet, dir=[0x200000007:0x1:0x0](ffff94d403679988) 00010000:00010000:3.0:1788572458.391133:0:67833:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030800/0x1e260726fbbc0a82 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4e03ef expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572458.391157:0:67833:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc0a89 lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xec901abfeb4e040b expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572458.391173:0:67833:0:(vvp_object.c:117:vvp_conf_set()) [0x20000a811:0x7:0x0]: losing layout lock 00010000:00010000:3.0:1788572458.391444:0:67833:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc0a89 lrc: 1/0,0 mode: --/PR res: [0x20000a811:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xec901abfeb4e040b expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572458.391453:0:67833:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030800/0x1e260726fbbc0a82 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4e03ef expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572458.391459:0:67833:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788572458.391462:0:67833:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031800/0x1e260726fbbc0a89 lrc: 0/0,0 mode: --/PR res: [0x20000a811:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xec901abfeb4e040b expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572458.391491:0:67833:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400030800/0x1e260726fbbc0a82 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4e03ef expref: -99 pid: 67833 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:3.0:1788572458.391511:0:67833:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572458.391519:0:67833:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c55880 x1875450488503552/t0(0) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 512/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572458.391541:0:67833:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:67833:1875450488503552:192.168.202.125@tcp:36:rm.0 00000100:00100000:3.0:1788572458.391609:0:67833:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408501840 going to sleep for 11 seconds 00000100:00100000:3.0:1788572458.395562:0:67833:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:67833:1875450488503552:192.168.202.125@tcp:36:rm.0 00000080:00200000:3.0:1788572458.395663:0:67833:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572458.395673:0:67833:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572458.395699:0:67833:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x20000a811:0x7:0x0](ffff94d410bcb248) 00010000:00010000:3.0:1788572458.629599:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572459.205902:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488503808:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572459.205992:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488504064:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572459.206023:0:2405:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:2.0:1788572459.206030:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488503680:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572459.206058:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488503936:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572459.206081:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488504192:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572459.213405:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488503680:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572459.213431:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488503936:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572459.213443:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488504192:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572459.213455:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488503808:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572459.213466:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488504064:192.168.202.125@tcp:400:kworker.0 00010000:00010000:3.0:1788572459.654953:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572460.677964:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572461.703025:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572462.727988:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572463.749457:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572464.328348:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488504320:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572464.328466:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488504576:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572464.328498:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488504832:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572464.328540:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:1.0:1788572464.328547:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488504448:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572464.328569:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488504704:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572464.339578:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488504704:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572464.339597:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488504576:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572464.339609:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488504832:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572464.353387:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488504448:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572464.362608:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488504320:192.168.202.125@tcp:400:kworker.0 00010000:00010000:3.0:1788572464.776904:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1788572465.598780:0:67899:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 00000080:00200000:0.0:1788572465.625977:0:67901:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:0.0:1788572465.626012:0:67901:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572465.626084:0:67901:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0a90 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 67901 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572465.626095:0:67901:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0a90 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 67901 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572465.626106:0:67901:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0a90 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 67901 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572465.626134:0:67901:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job dd:9bf287c0-f582-4b83-84c1-15e766218620:67901:1875450488504960:192.168.202.125@tcp:101:dd.0 00000100:00100000:0.0:1788572465.626253:0:67901:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4054113c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572465.636697:0:67901:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job dd:9bf287c0-f582-4b83-84c1-15e766218620:67901:1875450488504960:192.168.202.125@tcp:101:dd.0 00010000:00010000:0.0:1788572465.636728:0:67901:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0a90 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0435 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572465.636742:0:67901:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0a90 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0435 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572465.636754:0:67901:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0a90 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0435 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572465.636765:0:67901:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0a90 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0435 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572465.636778:0:67901:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0a90 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0435 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572465.636786:0:67901:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0a90 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0435 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572465.636794:0:67901:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0a90 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0435 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572465.636810:0:67901:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408d3ed80 x1875450488504960/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572481 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572465.636824:0:67901:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0a90 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0435 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572465.636838:0:67901:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788572465.636847:0:67901:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788572465.636873:0:67901:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1788572465.636879:0:67901:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0a90 00000080:00010000:0.0:1788572465.636885:0:67901:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572465.636888:0:67901:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a90 from it ffffb68a41fbfbf8 00010000:00010000:0.0:1788572465.636891:0:67901:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0a90 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0435 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572465.636900:0:67901:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0a90 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0435 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572465.636928:0:67901:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572465.636952:0:67901:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:0.0:1788572465.636967:0:67901:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572465.636977:0:67901:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0a90 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0435 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572465.636987:0:67901:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0a90 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0435 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572465.636998:0:67901:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572465.637002:0:67901:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572465.637005:0:67901:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0a90 00000080:00010000:0.0:1788572465.637008:0:67901:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a90 from it ffffb68a41fbfb38 00010000:00010000:0.0:1788572465.637010:0:67901:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0a90 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0435 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572465.637017:0:67901:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0a90 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0435 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788572465.637025:0:67901:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 3 00000080:00200000:0.0:1788572465.637030:0:67901:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f131a.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), file ffff94d408f30d80, open_flags 8241, mode 81b6 opened 0 00000080:00200000:0.0:1788572465.637039:0:67901:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f131a.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), intent=open|creat 00000002:00010000:0.0:1788572465.637062:0:67901:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f131a.replay-single,[0x20000a811:0x8:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0101102 00010000:00010000:0.0:1788572465.637074:0:67901:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0a90 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4e0435 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572465.637122:0:67901:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0a90 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4e0435 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572465.637129:0:67901:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572465.637132:0:67901:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0a90 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4e0435 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572465.637168:0:67901:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 67901 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572465.637559:0:67901:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 67901 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572465.637573:0:67901:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572465.637582:0:67901:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408d3ed80 x1875450488505216/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 752/66408 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572465.637593:0:67901:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 67901 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572465.637617:0:67901:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job dd:9bf287c0-f582-4b83-84c1-15e766218620:67901:1875450488505216:192.168.202.125@tcp:101:dd.0 00000100:00100000:0.0:1788572465.637771:0:67901:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4054113c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572465.643649:0:67901:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job dd:9bf287c0-f582-4b83-84c1-15e766218620:67901:1875450488505216:192.168.202.125@tcp:101:dd.0 00010000:00010000:0.0:1788572465.643679:0:67901:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PW ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572465.643713:0:67901:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 4/0,1 mode: --/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572465.643721:0:67901:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 4/0,1 mode: --/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572465.643732:0:67901:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 4/0,1 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572465.643744:0:67901:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 4/0,1 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572465.643762:0:67901:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 5/0,2 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572465.643771:0:67901:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 5/0,2 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572465.643778:0:67901:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 4/0,1 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572465.643792:0:67901:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x2b, status=0 req@ffff94d408d3ed80 x1875450488505216/t103079215113(103079215113) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 752/824 e 0 to 0 dl 1788572481 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788572465.643805:0:67901:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d408d3ed80 x1875450488505216/t103079215113(103079215113) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 752/824 e 0 to 0 dl 1788572481 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572465.643818:0:67901:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 216 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 3/0,1 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788572465.643829:0:67901:0:(mdc_locks.c:969:mdc_finish_enqueue()) ### DoM lock is returned by: open|creat, size: 0 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 3/0,1 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788572465.643840:0:67901:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 3/0,1 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572465.643851:0:67901:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 2 res 8589977617/6 (0 0) 00000080:00200000:0.0:1788572465.643857:0:67901:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788572465.643947:0:67901:0:(namei.c:147:ll_iget()) got inode: [0x20000a811:0x6:0x0](ffff94d4034963c8): rc = 0 00000080:00200000:0.0:1788572465.643968:0:67901:0:(file.c:6826:ll_layout_conf()) [0x20000a811:0x6:0x0]: layout version change: 4294967294 -> 2 00000080:00010000:0.0:1788572465.643980:0:67901:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a811:0x6:0x0](ffff94d4034963c8) for lock 0x1e260726fbbc0a97 00000080:00010000:0.0:1788572465.643994:0:67901:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a811:0x6:0x0](ffff94d4034963c8) 00000080:00200008:0.0:1788572465.644012:0:67901:0:(file.c:1090:ll_file_open()) START file f131a.replay-single:[0x20000a811:0x6:0x0](ffff94d4034963c8), flags 101101 00000002:00100000:0.0:1788572465.644038:0:67901:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d408d3ed80 x1875450488505216/t103079215113(103079215113) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 752/824 e 0 to 0 dl 1788572481 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788572465.644085:0:67901:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a97 from it ffff94d416ec4060 00010000:00010000:0.0:1788572465.644090:0:67901:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 3/0,1 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572465.644100:0:67901:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 2/0,0 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572465.644123:0:67901:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x20000a811:0x6:0x0](ffff94d4034963c8), inode mode 81a4 mask 42 00000002:00010000:0.0:1788572465.644150:0:67901:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x20000a811:0x6:0x0], intent: getxattr flags 0 00010000:00010000:0.0:1788572465.644191:0:67901:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166a00/0x1e260726fbbc0a9e lrc: 3/1,0 mode: --/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 67901 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572465.644198:0:67901:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166a00/0x1e260726fbbc0a9e lrc: 3/1,0 mode: --/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT flags: 0x0 pid: 67901 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572465.644231:0:67901:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166a00/0x1e260726fbbc0a9e lrc: 3/1,0 mode: --/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT flags: 0x0 pid: 67901 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572465.644254:0:67901:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job dd:9bf287c0-f582-4b83-84c1-15e766218620:67901:1875450488505344:192.168.202.125@tcp:101:dd.0 00000100:00100000:0.0:1788572465.644346:0:67901:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4054113c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572465.647018:0:67901:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job dd:9bf287c0-f582-4b83-84c1-15e766218620:67901:1875450488505344:192.168.202.125@tcp:101:dd.0 00010000:00010000:0.0:1788572465.647046:0:67901:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166a00/0x1e260726fbbc0a9e lrc: 4/1,0 mode: --/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0466 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572465.647065:0:67901:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166a00/0x1e260726fbbc0a9e lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0466 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572465.647076:0:67901:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166a00/0x1e260726fbbc0a9e lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0466 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572465.647102:0:67901:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff94d408d3e300 x1875450488505344/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 568/1272 e 0 to 0 dl 1788572481 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572465.647114:0:67901:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166a00/0x1e260726fbbc0a9e lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0466 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572465.647127:0:67901:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589977617/6 (0 0) 00000080:00200000:0.0:1788572465.647133:0:67901:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x20000a811:0x6:0x0] suppgids 0 -1: rc 0 00000080:00010000:0.0:1788572465.647179:0:67901:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a811:0x6:0x0](ffff94d4034963c8) for lock 0x1e260726fbbc0a9e 00000080:00010000:0.0:1788572465.647188:0:67901:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0a9e from it ffffb68a41fbfa38 00010000:00010000:0.0:1788572465.647193:0:67901:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166a00/0x1e260726fbbc0a9e lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0466 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572465.647224:0:67901:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166a00/0x1e260726fbbc0a9e lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0466 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1788572465.647262:0:67901:0:(llite_lib.c:2436:ll_setattr_raw()) START file f131a.replay-single:[0x20000a811:0x6:0x0](ffff94d4034963c8) current size 0, valid attrs a068, mode 0, uid 2, gid 0, new size 0, atime 1788572465.000000000, mtime 1788572465.000000000, ctime 1788572465.000000000 00000020:00100000:0.0:1788572465.647317:0:67901:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788572465.647324:0:67901:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408d3e300 x1875450488505600/t0(0) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572465.647352:0:67901:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job dd:9bf287c0-f582-4b83-84c1-15e766218620:67901:1875450488505600:192.168.202.125@tcp:36:dd.0 00000100:00100000:0.0:1788572465.647426:0:67901:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4054113c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572465.649689:0:2396:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.125@tcp (source: 12345-192.168.202.125@tcp) 00000100:00100000:3.0:1788572465.649839:0:42439:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875450511481984 00000100:00100000:3.0:1788572465.649850:0:42439:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875450511481984 00000100:00100000:3.0:1788572465.649907:0:42439:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.125@tcp, seq: 92 00000100:00100000:3.0:1788572465.649918:0:42439:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff94d408c62300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:9bf287c0-f582-4b83-84c1-15e766218620+5:66200:x1875450511481984:12345-192.168.202.125@tcp:104: 00010000:00010000:3.0:1788572465.649938:0:42439:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 2/0,0 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x5b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572465.650009:0:42439:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788572465.650016:0:42439:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff94d408c62300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:9bf287c0-f582-4b83-84c1-15e766218620+5:66200:x1875450511481984:12345-192.168.202.125@tcp:104: Request processed in 101us (335us total) trans 0 rc 0/0 00000100:00100000:3.0:1788572465.650023:0:42439:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.125@tcp, seq: 92 00010000:00010000:0.0:1788572465.650036:0:42449:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788572465.650042:0:42449:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 2/0,0 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x5b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572465.650051:0:42449:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412166e00 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:0.0:1788572465.650069:0:42449:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 2/0,0 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x5b/0x2 rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572465.650080:0:42449:0:(ldlm_request.c:1334:ldlm_cli_convert_req()) ### client-side convert ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 2/0,0 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x5b/0x2 rrc: 3 type: IBT gid 0 flags: 0x420402000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1788572465.650160:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488505728:192.168.202.125@tcp:102:ldlm_bl.0 00010000:00010000:0.0:1788572465.650288:0:42449:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 2/0,0 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x2 rrc: 3 type: IBT gid 0 flags: 0x10420402000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572465.650298:0:42449:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 2/0,0 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572465.650305:0:42449:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 2/0,0 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572465.650314:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572465.650315:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572465.650317:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572465.654146:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488505728:192.168.202.125@tcp:102:ldlm_bl.0 00000100:00100000:0.0:1788572465.655574:0:67901:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job dd:9bf287c0-f582-4b83-84c1-15e766218620:67901:1875450488505600:192.168.202.125@tcp:36:dd.0 00000080:00200000:0.0:1788572465.655658:0:67901:0:(file.c:1848:ll_io_set_mirror()) f131a.replay-single: desiginated mirror: 0 00000080:00200000:0.0:1788572465.655665:0:67901:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572465.655682:0:67901:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:0.0:1788572465.655709:0:67901:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x0 00000080:00200000:0.0:1788572465.655713:0:67901:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000002:00010000:0.0:1788572465.655724:0:67901:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 4 00000002:00010000:0.0:1788572465.655732:0:67901:0:(mdc_dev.c:1007:mdc_lock_init()) ### lock ffff94d41257df30, mdc lock ffff94d41242a4c0, flags 80000000 00010000:00010000:0.0:1788572465.655756:0:67901:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 2/0,1 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572465.655773:0:67901:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 2/0,1 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572465.655787:0:67901:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 6/0,2 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000002:00010000:0.0:1788572465.655797:0:67901:0:(mdc_dev.c:456:mdc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=0 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 5/0,2 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572465.655809:0:67901:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 6/0,2 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x0 rrc: 3 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572465.655817:0:67901:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 5/0,1 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x0 rrc: 3 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572465.655840:0:67901:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 4/0,1 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x0 rrc: 3 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000100:00100000:1.0:1788572465.656140:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488505856:192.168.202.125@tcp:10:dd.0 00000100:00100000:1.0:1788572465.661080:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488505856:192.168.202.125@tcp:10:dd.0 00000080:00200000:0.0:1788572465.661297:0:67901:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000a811:0x6:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1788572465.661305:0:67901:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff94d405f9ee48 (ffff94d405fa02d8/0) at cl_lock_release():245 00000020:00010000:0.0:1788572465.661309:0:67901:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d405f9ee48 (ffff94d405fa02d8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788572465.661315:0:67901:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d41257df30 (ffff94d405fa0998/1) at cl_lock_cancel():142 00010000:00010000:0.0:1788572465.661327:0:67901:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 4/0,1 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x0 rrc: 3 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572465.661341:0:67901:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 3/0,0 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x0 rrc: 3 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00010000:0.0:1788572465.661357:0:67901:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d405f9ee48 (ffff94d405fa02d8/0) at cl_lock_fini():72 00000020:00010000:0.0:1788572465.661360:0:67901:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d41257df30 (ffff94d405fa02d8/1) at cl_lock_fini():72 00000080:00200000:0.0:1788572465.661611:0:67901:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x6:0x0] ignore/verify layout 0/0, layout version 2 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1788572465.661634:0:67901:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file f131a.replay-single:[0x20000a811:0x6:0x0](ffff94d4034963c8) current size 0, valid attrs 1000a168, mode 0, uid 2, gid 0, new size 0, atime 1788572465.000000000, mtime 1788572465.000000000, ctime 1788572465.000000000, rc 0 00000080:00200008:0.0:1788572465.661747:0:67901:0:(file.c:2640:do_file_write_iter()) START file f131a.replay-single:[0x20000a811:0x6:0x0], ppos: 0, count: 8 00000080:00200000:0.0:1788572465.661760:0:67901:0:(rw26.c:675:ll_write_begin()) Writing 0 of 0 to 8 bytes 00000080:00200000:0.0:1788572465.661767:0:67901:0:(file.c:2616:ll_do_tiny_write()) result: 0, original count 8 00000080:00200000:0.0:1788572465.661793:0:67901:0:(file.c:2090:ll_file_io_generic()) f131a.replay-single: write ppos: 0, bytes: 8 00000080:00200000:0.0:1788572465.661798:0:67901:0:(file.c:1848:ll_io_set_mirror()) f131a.replay-single: desiginated mirror: 0 00000020:00200000:0.0:1788572465.661805:0:67901:0:(cl_io.c:214:cl_io_rw_init()) header@00000000386dd75d[0x110, 1, [0x20000a811:0x6:0x0] hash] 00000020:00200000:0.0:1788572465.661807:0:67901:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 8) 0 0 00000080:00200000:0.0:1788572465.661811:0:67901:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1788572465.661824:0:67901:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:0.0:1788572465.661831:0:67901:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 8) 8, 8 00020000:00200000:0.0:1788572465.661834:0:67901:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572465.661861:0:67901:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000002:00010000:0.0:1788572465.661872:0:67901:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000002:00010000:0.0:1788572465.661878:0:67901:0:(mdc_dev.c:1007:mdc_lock_init()) ### lock ffff94d41257df30, mdc lock ffff94d41242a4c0, flags 0 00010000:00010000:0.0:1788572465.661897:0:67901:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 2/0,1 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x0 rrc: 4 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572465.661910:0:67901:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 2/0,1 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x0 rrc: 3 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572465.661922:0:67901:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 6/0,2 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x0 rrc: 3 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572465.661930:0:67901:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 6/0,2 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x0 rrc: 3 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572465.661937:0:67901:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 5/0,1 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x0 rrc: 3 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788572465.661949:0:67901:0:(vvp_io.c:1292:vvp_io_write_start()) f131a.replay-single: write [0, 8) append 0 append lockpos 0 00000080:00200000:0.0:1788572465.661957:0:67901:0:(rw26.c:675:ll_write_begin()) Writing 0 of 0 to 8 bytes 00000080:00200000:0.0:1788572465.661995:0:67901:0:(rw26.c:869:ll_write_end()) pos 0, len 8, copied 8 00000080:00200000:0.0:1788572465.662002:0:67901:0:(rw26.c:927:ll_write_end()) page@00000000c11d2f2d[3 00000000551cd2bc 2 1 00000000af89aa6c 0 0 0] 00000080:00200000:0.0:1788572465.662006:0:67901:0:(rw26.c:927:ll_write_end()) vmpage @00000000f53c0f09 17ffffc0001001 3:0 ffff94d416eec7e0 0 lru 00000080:00200000:0.0:1788572465.662014:0:67901:0:(rw26.c:927:ll_write_end()) osc-page@000000007f590046 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 000000009fcc4f63 00000000eb5518d7 > 3< 0 0 > 4< 0 0 8 2097152 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1788572465.662016:0:67901:0:(rw26.c:927:ll_write_end()) end page@00000000c11d2f2d 00000080:00200000:0.0:1788572465.662017:0:67901:0:(rw26.c:927:ll_write_end()) queued cl_page: 1. 00000080:00200000:0.0:1788572465.662022:0:67901:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 8 prio 0 00000008:00010000:0.0:1788572465.662044:0:67901:0:(osc_cache.c:857:osc_extent_find()) ### extent: 000000008b6de3dd ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 4/0,1 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x0 rrc: 3 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788572465.662061:0:67901:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff94d403496578, count 1, dirtied 1 00000080:00200000:0.0:1788572465.662065:0:67901:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 8 bytes, tot: 8 00000080:00200000:0.0:1788572465.662068:0:67901:0:(vvp_io.c:1358:vvp_io_write_start()) f131a.replay-single: write bytes 0, result: 8 00000020:00010000:0.0:1788572465.662079:0:67901:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff94d405f9ee48 (ffff94d405fa02d8/0) at cl_lock_release():245 00000020:00010000:0.0:1788572465.662082:0:67901:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d405f9ee48 (ffff94d405fa02d8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788572465.662085:0:67901:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d41257df30 (ffff94d405fa0998/1) at cl_lock_cancel():142 00010000:00010000:0.0:1788572465.662100:0:67901:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 5/0,1 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x0 rrc: 3 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572465.662109:0:67901:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 4/0,0 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x0 rrc: 3 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000020:00010000:0.0:1788572465.662119:0:67901:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d405f9ee48 (ffff94d405fa02d8/0) at cl_lock_fini():72 00000020:00010000:0.0:1788572465.662121:0:67901:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d41257df30 (ffff94d405fa02d8/1) at cl_lock_fini():72 00000080:00200000:0.0:1788572465.662127:0:67901:0:(vvp_io.c:533:vvp_io_advance()) advancing 8 bytes 00000080:00200000:0.0:1788572465.662131:0:67901:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:0.0:1788572465.662146:0:67901:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x6:0x0] ignore/verify layout 0/0, layout version 2 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572465.662150:0:67901:0:(file.c:2238:ll_file_io_generic()) f131a.replay-single: 2 io complete with rc: 0, result: 8, restart: 0 00000080:00200000:0.0:1788572465.662154:0:67901:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000a811:0x6:0x0] updating i_size 8 i_blocks 0 00000080:00200000:0.0:1788572465.662158:0:67901:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 8 00000080:00200008:0.0:1788572465.663589:0:67901:0:(file.c:479:ll_file_release()) START file f131a.replay-single:[0x20000a811:0x6:0x0](ffff94d4034963c8), flags 100001 00000002:00100000:0.0:1788572465.663669:0:67901:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff94d408d3ed80 x1875450488505216/t103079215113(103079215113) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 752/824 e 0 to 0 dl 1788572481 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788572465.663684:0:67901:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572465.663690:0:67901:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408d3dc00 x1875450488506112/t0(0) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/712 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788572465.663717:0:67901:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job dd:9bf287c0-f582-4b83-84c1-15e766218620:67901:1875450488506112:192.168.202.125@tcp:35:dd.0 00000100:00100000:0.0:1788572465.663820:0:67901:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4054113c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572465.667848:0:67901:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job dd:9bf287c0-f582-4b83-84c1-15e766218620:67901:1875450488506112:192.168.202.125@tcp:35:dd.0 00000002:00100000:0.0:1788572465.667888:0:67901:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff94d408d3ed80 x1875450488505216/t103079215113(103079215113) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 752/824 e 0 to 0 dl 1788572481 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1788572465.812481:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572466.822549:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572467.845837:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572468.869576:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572469.446632:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488506240:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572469.446650:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488506368:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572469.446728:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488506496:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572469.454743:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488506624:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572469.463155:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488506752:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572469.463700:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488506624:192.168.202.125@tcp:400:kworker.0 00000100:00080000:1.0:1788572469.463732:0:2404:0:(recover.c:205:ptlrpc_request_handle_notconn()) import lustre-MDT0000-mdc-ffff94d405cc3800 of lustre-MDT0000_UUID@192.168.202.125@tcp abruptly disconnected: reconnecting 00000100:00080000:1.0:1788572469.463737:0:2404:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:1.0:1788572469.463741:0:2404:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff94d405cc3800: Connection to lustre-MDT0000 (at 192.168.202.125@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:1.0:1788572469.493820:0:2404:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572469.493836:0:2404:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3582 00000100:00080000:1.0:1788572469.493854:0:2404:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572469.493866:0:2404:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1788572469.493902:0:2404:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488506880/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572469.493930:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488506368:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572469.498298:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488506752:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572469.498361:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488506496:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572469.498490:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488506880:192.168.202.125@tcp:38:ptlrpcd_00_01.0 00000100:00080000:0.0:1788572469.500590:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572469.500605:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-19) 00000100:00100000:0.0:1788572469.500611:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488506880:192.168.202.125@tcp:38:ptlrpcd_00_01.0 00010000:00010000:3.0:1788572469.896463:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572470.920243:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1788572470.985360:0:470:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572470.985378:0:470:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:2.0:1788572470.985400:0:470:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x0 00000008:00010000:2.0:1788572470.985421:0:470:0:(osc_cache.c:2022:try_to_add_extent_for_io()) ### extent: 000000008b6de3dd ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 2/0,0 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x0 rrc: 3 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000008:00010000:2.0:1788572470.985441:0:470:0:(osc_cache.c:1176:osc_extent_make_ready()) ### extent: 000000008b6de3dd ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 2/0,0 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x0 rrc: 3 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572470.985496:0:470:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 3/0,0 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x0 rrc: 3 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00000008:00100000:2.0:1788572470.985603:0:470:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff94d408c63800 - object 0x20000a811:6 offset 0<>8 00000100:00080000:3.0:1788572470.985812:0:2405:0:(client.c:1832:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff94d408c63800 x1875450488507008/t0(0) o4->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:13/10 lens 496/448 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788572470.985831:0:2405:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00080000:3.0:1788572470.985837:0:2405:0:(client.c:1832:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff94d408c60380 x1875450488507136/t0(0) o16->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 440/432 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1788572471.946390:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572472.965414:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572473.989520:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572474.566773:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572474.566781:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572474.566785:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572474.566795:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3622 00000100:00080000:3.0:1788572474.566806:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1788572474.566814:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572474.566824:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1788572474.566845:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488507392/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572474.566915:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488507520:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572474.566921:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488507264:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572474.566992:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488507392:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788572474.569371:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488507648:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572474.580705:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488507776:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572474.582585:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488507648:192.168.202.125@tcp:400:kworker.0 00000100:00080000:0.0:1788572474.582795:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572474.582803:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-19) 00000100:00100000:0.0:1788572474.582807:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488507392:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788572474.588637:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488507776:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572474.590079:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488507520:192.168.202.125@tcp:400:kworker.0 00010000:00010000:1.0:1788572475.022357:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572476.037981:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572477.062897:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572478.089681:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572479.112135:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572479.685418:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572479.685425:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572479.685429:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572479.685439:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3627 00000100:00080000:3.0:1788572479.685450:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1788572479.685458:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572479.685469:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1788572479.685490:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488508032/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572479.685543:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405bad000 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00080000:3.0:1788572479.685547:0:66171:0:(import.c:2101:ptlrpc_disconnect_and_idle_import()) lustre-OST0000-osc-ffff94d405cc3800: disconnect after 21s idle 00000100:00080000:3.0:1788572479.685557:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405bab800 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:00080000:3.0:1788572479.685559:0:66171:0:(import.c:2101:ptlrpc_disconnect_and_idle_import()) lustre-OST0001-osc-ffff94d405cc3800: disconnect after 21s idle 00000100:00100000:3.0:1788572479.685603:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488508160:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572479.685713:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788572479.685721:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488507904:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572479.686190:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488508032:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572479.686265:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488508288:192.168.202.125@tcp:9:kworker.0 00000100:00100000:0.0:1788572479.686290:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488508416:192.168.202.125@tcp:9:kworker.0 00000100:00080000:0.0:1788572479.692402:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572479.692420:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-19) 00000100:00100000:0.0:1788572479.692426:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488508032:192.168.202.125@tcp:38:kworker.0 00000100:00080000:0.0:1788572479.692453:0:2402:0:(import.c:2011:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff94d408c54000 x1875450488508288/t0(0) o9->lustre-OST0000-osc-ffff94d405cc3800@192.168.202.125@tcp:28/4 lens 224/224 e 0 to 0 dl 1788572489 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572479.692463:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bad000 lustre-OST0000_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:0.0:1788572479.692466:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488508288:192.168.202.125@tcp:9:kworker.0 00000100:00080000:0.0:1788572479.692479:0:2402:0:(import.c:2011:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff94d408c55f80 x1875450488508416/t0(0) o9->lustre-OST0001-osc-ffff94d405cc3800@192.168.202.125@tcp:28/4 lens 224/224 e 0 to 0 dl 1788572489 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572479.692486:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab800 lustre-OST0001_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:0.0:1788572479.692490:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488508416:192.168.202.125@tcp:9:kworker.0 00000100:00100000:3.0:1788572479.696197:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488508160:192.168.202.125@tcp:400:kworker.0 00010000:00010000:3.0:1788572480.135005:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572481.157782:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572482.181485:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572483.205931:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572484.229720:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572484.808778:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572484.808787:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572484.808791:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572484.808802:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3633 00000100:00080000:2.0:1788572484.808814:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1788572484.808822:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572484.808832:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1788572484.808853:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488508672/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572484.809075:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488508672:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788572484.812349:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488508544:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572484.812469:0:2405:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788572484.812478:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488508800:192.168.202.125@tcp:400:kworker.0 00000100:00080000:0.0:1788572484.813140:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572484.813149:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-19) 00000100:00100000:0.0:1788572484.813153:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488508672:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788572484.820185:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488508800:192.168.202.125@tcp:400:kworker.0 00010000:00010000:1.0:1788572485.253895:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1788572485.829573:0:2403:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572469/real 1788572469] req@ffff94d408d3dc00 x1875450488506240/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572485 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572485.829611:0:2403:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff94d408d3dc00 x1875450488506240/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572485 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572485.829631:0:2403:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from FULL to DISCONN 00000100:02020000:0.0:1788572485.829636:0:2403:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.202.125@tcp: Connection to MGS (at 192.168.202.125@tcp) was lost; in progress operations using this service will fail 00000100:00080000:0.0:1788572485.829640:0:2403:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.202.125@tcp for MGC192.168.202.125@tcp not replayable, auto-deactivating 00000100:00080000:0.0:1788572485.829642:0:2403:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:0.0:1788572485.829646:0:2403:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff94d408d3dc00 x1875450488506240/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572485 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572485.829657:0:2403:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff94d408c55c00 x1875450488507264/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572490 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572485.829678:0:2403:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff94d408c56300 x1875450488507904/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572495 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572485.829696:0:2403:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff94d408c61880 x1875450488508544/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572500 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:0.0:1788572485.829706:0:2403:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:0.0:1788572485.829709:0:2403:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:0.0:1788572485.829710:0:2403:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:0.0:1788572485.829712:0:2403:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00100000:0.0:1788572485.829744:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488506240:192.168.202.125@tcp:400:kworker.0 00000100:00080000:1.0:1788572485.829831:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) e60431de-58b9-4f69-9048-11a3d82ddcdd->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572485.829839:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572485.829842:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572485.829847:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3580 00000100:00080000:1.0:1788572485.829862:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d40490a800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788572485.829899:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488508928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572485.829925:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572485.829929:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572485.829930:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572485.829934:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3638 00000100:00080000:1.0:1788572485.829937:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572485.829941:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572485.829950:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1788572485.829961:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488509056/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572485.830005:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488509184:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572485.830158:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488507904:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572485.830342:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488508544:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572485.830352:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488507264:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572485.830826:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e60431de-58b9-4f69-9048-11a3d82ddcdd:2402:1875450488508928:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788572485.830888:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488509056:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572485.844481:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488509184:192.168.202.125@tcp:400:kworker.0 00000100:00080000:0.0:1788572485.845425:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572485.845435:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-19) 00000100:00100000:0.0:1788572485.845439:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488509056:192.168.202.125@tcp:38:kworker.0 00010000:00010000:3.0:1788572486.278923:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572487.301835:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572488.325639:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572489.353249:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572490.374147:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572490.951496:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) e60431de-58b9-4f69-9048-11a3d82ddcdd->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572490.951507:0:66171:0:(pinger.c:247:ptlrpc_pinger_process_import()) e60431de-58b9-4f69-9048-11a3d82ddcdd->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788572490.951510:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572490.951513:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572490.951518:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572490.951529:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3639 00000100:00080000:1.0:1788572490.951541:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572490.951549:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572490.951560:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1788572490.951597:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488509312/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572490.951685:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488509440:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572490.951872:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488509312:192.168.202.125@tcp:38:kworker.0 00000100:00080000:0.0:1788572490.955893:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572490.955909:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-19) 00000100:00100000:0.0:1788572490.955915:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488509312:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572490.956377:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488509440:192.168.202.125@tcp:400:kworker.0 00010000:00010000:1.0:1788572491.398834:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572492.421515:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572493.445484:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572494.473133:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572495.045784:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572485/real 1788572485] req@ffff94d4049bf800 x1875450488508928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572495 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572495.045822:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488508928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572495 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572495.045841:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572495.045865:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788572495.045870:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e60431de-58b9-4f69-9048-11a3d82ddcdd:2402:1875450488508928:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788572495.046012:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) e60431de-58b9-4f69-9048-11a3d82ddcdd->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572495.046017:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572495.046020:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572495.046025:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3639 00000100:00080000:3.0:1788572495.046034:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1788572495.046039:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d40490a800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788572495.046065:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488509568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572495.046091:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572495.046095:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572495.046096:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572495.046099:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3644 00000100:00080000:3.0:1788572495.046101:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572495.046104:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572495.046111:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1788572495.046118:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488509696/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572495.046153:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488509824:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572495.046536:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e60431de-58b9-4f69-9048-11a3d82ddcdd:2402:1875450488509568:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788572495.046580:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488509696:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788572495.056687:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488509824:192.168.202.125@tcp:400:kworker.0 00000100:00080000:2.0:1788572495.061499:0:2402:0:(import.c:1132:ptlrpc_connect_interpret()) MGC192.168.202.125@tcp: connect to target with instance 0 10000000:01000000:2.0:1788572495.061518:0:2402:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:02000400:2.0:1788572495.061526:0:2402:0:(import.c:1284:ptlrpc_connect_interpret()) Evicted from MGS (at 192.168.202.125@tcp) after server handle changed from 0xec901abfeb4dfee7 to 0xec901abfeb4e0490 00000100:00080000:2.0:1788572495.061531:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from CONNECTING to EVICTED 00000100:00080000:2.0:1788572495.061536:0:2402:0:(import.c:1674:ptlrpc_import_recovery_state_machine()) evicted from MGS@192.168.202.125@tcp; invalidating 00000100:00100000:2.0:1788572495.062454:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e60431de-58b9-4f69-9048-11a3d82ddcdd:2402:1875450488509568:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788572495.062892:0:68124:0:(import.c:1598:ptlrpc_invalidate_import_thread()) thread invalidate import MGC192.168.202.125@tcp to MGS@192.168.202.125@tcp 00000100:00100000:3.0:1788572495.062899:0:68124:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:3.0:1788572495.062902:0:68124:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00010000:00010000:3.0:1788572495.062919:0:68124:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.125@tcp lock: ffff94d412166200/0x1e260726fbbc0a35 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110e400000000 nid: local remote: 0xec901abfeb4e0092 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788572495.062930:0:68124:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.125@tcp lock: ffff94d412166200/0x1e260726fbbc0a35 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110f400000000 nid: local remote: 0xec901abfeb4e0092 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572495.062936:0:68124:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:3.0:1788572495.062956:0:68124:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572495.062966:0:68124:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.125@tcp lock: ffff94d412166200/0x1e260726fbbc0a35 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x590f400000000 nid: local remote: 0xec901abfeb4e0092 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572495.062980:0:68124:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.125@tcp lock: ffff94d400031600/0x1e260726fbbc0a43 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0xec901abfeb4e00df expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788572495.062987:0:68124:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.125@tcp lock: ffff94d400031600/0x1e260726fbbc0a43 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0xec901abfeb4e00df expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572495.062992:0:68124:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:3.0:1788572495.062995:0:68124:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572495.062998:0:68124:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.125@tcp lock: ffff94d400031600/0x1e260726fbbc0a43 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0xec901abfeb4e00df expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572495.063013:0:68124:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.125@tcp lock: ffff94d412166800/0x1e260726fbbc0a2e lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0xec901abfeb4e007d expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788572495.063020:0:68124:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.125@tcp lock: ffff94d412166800/0x1e260726fbbc0a2e lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0xec901abfeb4e007d expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572495.063026:0:68124:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:3.0:1788572495.063029:0:68124:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572495.063032:0:68124:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.125@tcp lock: ffff94d412166800/0x1e260726fbbc0a2e lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0xec901abfeb4e007d expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572495.063042:0:68124:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.125@tcp lock: ffff94d400031c00/0x1e260726fbbc0a3c lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0xec901abfeb4e00b5 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788572495.063048:0:68124:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.125@tcp lock: ffff94d400031c00/0x1e260726fbbc0a3c lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0xec901abfeb4e00b5 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572495.063053:0:68124:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 00010000:00010000:3.0:1788572495.063056:0:68124:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572495.063059:0:68124:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.125@tcp lock: ffff94d400031c00/0x1e260726fbbc0a3c lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0xec901abfeb4e00b5 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:3.0:1788572495.063074:0:68124:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from EVICTED to RECOVER 00000100:00080000:3.0:1788572495.063079:0:68124:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from RECOVER to FULL 10000000:01000000:3.0:1788572495.063149:0:68124:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:3.0:1788572495.063154:0:68124:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:02000000:3.0:1788572495.063159:0:68124:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) MGC192.168.202.125@tcp: Connection restored to 192.168.202.125@tcp (at 192.168.202.125@tcp) 00000100:00080000:0.0:1788572495.083015:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572495.083030:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-19) 00000100:00100000:0.0:1788572495.083035:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488509696:192.168.202.125@tcp:38:kworker.0 00010000:00010000:3.0:1788572495.493791:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572496.517437:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788572496.581350:0:42451:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788572496.581357:0:42451:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-cliir-ffff94d405cc3800 from 109 10000000:01000000:2.0:1788572496.581363:0:42451:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788572496.581415:0:42451:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412537000/0x1e260726fbbc0aa5 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.581429:0:42451:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.125@tcp lock: ffff94d412537000/0x1e260726fbbc0aa5 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.581440:0:42451:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.125@tcp lock: ffff94d412537000/0x1e260726fbbc0aa5 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572496.581482:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488509952:192.168.202.125@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788572496.581580:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4185f4240 going to sleep for 11 seconds 00000100:00100000:0.0:1788572496.600817:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488509952:192.168.202.125@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788572496.600850:0:42451:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.125@tcp lock: ffff94d412537000/0x1e260726fbbc0aa5 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e05b6 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572496.600864:0:42451:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.125@tcp lock: ffff94d412537000/0x1e260726fbbc0aa5 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xec901abfeb4e05b6 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572496.600956:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488510080:192.168.202.125@tcp:256:ll_cfg_requeue.0 00000100:00100000:0.0:1788572496.601044:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405411180 going to sleep for 11 seconds 00000100:00100000:0.0:1788572496.618060:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488510080:192.168.202.125@tcp:256:ll_cfg_requeue.0 00000100:00080000:0.0:1788572496.618126:0:42451:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff94d405bab000 already not connected (conn 88, was 0): DISCONN 00010000:00080000:0.0:1788572496.618138:0:42451:0:(ldlm_lib.c:85:import_set_conn()) imp ffff94d405bab000@lustre-MDT0000-mdc-ffff94d405cc3800: found existing conn 192.168.202.125@tcp, moved to head 00000100:00080000:0.0:1788572496.618144:0:42451:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572496.618149:0:42451:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 0 00000100:00080000:0.0:1788572496.618159:0:42451:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572496.618169:0:42451:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1788572496.618234:0:42451:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488510208/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 10000000:01000000:0.0:1788572496.618298:0:42451:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.202.125@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:0.0:1788572496.618308:0:42451:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412537000/0x1e260726fbbc0aa5 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e05b6 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572496.618319:0:42451:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.125@tcp lock: ffff94d412537000/0x1e260726fbbc0aa5 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e05b6 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788572496.618328:0:42451:0:(mgc_request.c:611:do_requeue()) updating log lustre-client 10000000:01000000:0.0:1788572496.618332:0:42451:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-client-ffff94d405cc3800 from 52 10000000:01000000:0.0:1788572496.618335:0:42451:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:0.0:1788572496.618361:0:42451:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412167e00/0x1e260726fbbc0aac lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572496.618370:0:42451:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.125@tcp lock: ffff94d412167e00/0x1e260726fbbc0aac lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572496.618381:0:42451:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.125@tcp lock: ffff94d412167e00/0x1e260726fbbc0aac lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572496.618400:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488510336:192.168.202.125@tcp:101:ll_cfg_requeue.0 00000100:00100000:0.0:1788572496.618488:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405411180 going to sleep for 11 seconds 00000100:00100000:2.0:1788572496.619199:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488510208:192.168.202.125@tcp:38:ll_cfg_requeue.0 00010000:00080000:2.0:1788572496.624836:0:2402:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff94d408d3f480 x1875450488510208/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/416 e 0 to 0 dl 1788572506 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572496.624862:0:2402:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to target with instance 25 00000100:00080000:2.0:1788572496.624883:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:2.0:1788572496.624890:0:2402:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788572496.624893:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 103079215110 gen 4 00000100:00100000:2.0:1788572496.624900:0:2402:0:(client.c:3156:ptlrpc_free_committed()) @@@ stopping search req@ffff94d408d3c700 x1875450488503040/t103079215111(103079215111) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572474 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572496.624909:0:2402:0:(recover.c:53:ptlrpc_replay_next()) import ffff94d405bab000 from lustre-MDT0000_UUID committed 103079215110 last 0 00000100:00080000:2.0:1788572496.624918:0:2402:0:(client.c:3540:ptlrpc_replay_req()) @@@ REPLAY req@ffff94d408d3c700 x1875450488503040/t103079215111(103079215111) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572474 ref 1 fl New:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572496.624932:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488510208:192.168.202.125@tcp:38:ll_cfg_requeue.0 00000100:00100000:2.0:1788572496.624968:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488503040:192.168.202.125@tcp:36:mcreate.0 00000100:00100000:0.0:1788572496.630800:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488510336:192.168.202.125@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788572496.630841:0:42451:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.125@tcp lock: ffff94d412167e00/0x1e260726fbbc0aac lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e05bd expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572496.630857:0:42451:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.125@tcp lock: ffff94d412167e00/0x1e260726fbbc0aac lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xec901abfeb4e05bd expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572496.630962:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488510464:192.168.202.125@tcp:501:ll_cfg_requeue.0 00000100:00100000:0.0:1788572496.631068:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405411180 going to sleep for 11 seconds 00000100:00100000:0.0:1788572496.636421:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488510464:192.168.202.125@tcp:501:ll_cfg_requeue.0 00000100:00100000:0.0:1788572496.636503:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488510592:192.168.202.125@tcp:503:ll_cfg_requeue.0 00000100:00100000:0.0:1788572496.636617:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405411180 going to sleep for 11 seconds 00000100:00100000:0.0:1788572496.643267:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488510592:192.168.202.125@tcp:503:ll_cfg_requeue.0 00000040:00080000:0.0:1788572496.644464:0:68134:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 64768 count 51 00000020:01000000:2.0:1788572496.645309:0:42451:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 52-51 (rc=0) 10000000:01000000:2.0:1788572496.645335:0:42451:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.202.125@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:2.0:1788572496.645349:0:42451:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412167e00/0x1e260726fbbc0aac lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e05bd expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.645361:0:42451:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.125@tcp lock: ffff94d412167e00/0x1e260726fbbc0aac lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e05bd expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572496.645371:0:42451:0:(mgc_request.c:611:do_requeue()) updating log params 10000000:01000000:2.0:1788572496.645374:0:42451:0:(mgc_request.c:1801:mgc_process_log()) Process log params-ffff94d405cc3800 from 7 10000000:01000000:2.0:1788572496.645377:0:42451:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:2.0:1788572496.645421:0:42451:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412536000/0x1e260726fbbc0ab3 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.645429:0:42451:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.125@tcp lock: ffff94d412536000/0x1e260726fbbc0ab3 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.645446:0:42451:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.125@tcp lock: ffff94d412536000/0x1e260726fbbc0ab3 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572496.645474:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488510720:192.168.202.125@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788572496.645575:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4185f4840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572496.659385:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488510720:192.168.202.125@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788572496.659419:0:42451:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.125@tcp lock: ffff94d412536000/0x1e260726fbbc0ab3 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e05c4 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.659433:0:42451:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.125@tcp lock: ffff94d412536000/0x1e260726fbbc0ab3 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xec901abfeb4e05c4 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572496.659521:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488510848:192.168.202.125@tcp:501:ll_cfg_requeue.0 00000100:00100000:2.0:1788572496.659623:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4185f4840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572496.672755:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488510848:192.168.202.125@tcp:501:ll_cfg_requeue.0 00000100:00100000:2.0:1788572496.672827:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488510976:192.168.202.125@tcp:503:ll_cfg_requeue.0 00000100:00100000:2.0:1788572496.672935:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4185f4840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572496.678641:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488510976:192.168.202.125@tcp:503:ll_cfg_requeue.0 00000040:00080000:2.0:1788572496.680871:0:68135:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 64768 count 7 00000020:01000000:0.0:1788572496.690274:0:42451:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 7-6 (rc=0) 10000000:01000000:0.0:1788572496.690312:0:42451:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.202.125@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:0.0:1788572496.690325:0:42451:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412536000/0x1e260726fbbc0ab3 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e05c4 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572496.690339:0:42451:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.125@tcp lock: ffff94d412536000/0x1e260726fbbc0ab3 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e05c4 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788572496.690351:0:42451:0:(mgc_request.c:611:do_requeue()) updating log lustre-sptlrpc 10000000:01000000:0.0:1788572496.690354:0:42451:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-sptlrpc-ffff94d4113f0000 from 1 10000000:01000000:0.0:1788572496.690357:0:42451:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:0.0:1788572496.690401:0:42451:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412167800/0x1e260726fbbc0aba lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572496.690410:0:42451:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.125@tcp lock: ffff94d412167800/0x1e260726fbbc0aba lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572496.690422:0:42451:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.125@tcp lock: ffff94d412167800/0x1e260726fbbc0aba lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572496.690450:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488511104:192.168.202.125@tcp:101:ll_cfg_requeue.0 00000100:00100000:0.0:1788572496.690554:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405411180 going to sleep for 11 seconds 00000100:00100000:0.0:1788572496.698418:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488511104:192.168.202.125@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788572496.698452:0:42451:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.125@tcp lock: ffff94d412167800/0x1e260726fbbc0aba lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e05cb expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572496.698521:0:42451:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.125@tcp lock: ffff94d412167800/0x1e260726fbbc0aba lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xec901abfeb4e05cb expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572496.700403:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488511232:192.168.202.125@tcp:501:ll_cfg_requeue.0 00000100:00100000:0.0:1788572496.700691:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405411180 going to sleep for 11 seconds 00000100:00100000:0.0:1788572496.710626:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488511232:192.168.202.125@tcp:501:ll_cfg_requeue.0 10000000:01000000:0.0:1788572496.710678:0:42451:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.202.125@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:0.0:1788572496.710689:0:42451:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412167800/0x1e260726fbbc0aba lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e05cb expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572496.710702:0:42451:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.125@tcp lock: ffff94d412167800/0x1e260726fbbc0aba lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e05cb expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572497.543086:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572498.566099:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572499.589639:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572500.165405:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 1 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572500.165411:0:66171:0:(pinger.c:247:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:3.0:1788572500.165560:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488511360:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572500.165680:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788572500.165688:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488511488:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572500.179320:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488511488:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572500.179339:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488511360:192.168.202.125@tcp:400:kworker.0 00010000:00010000:1.0:1788572500.613791:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572501.632525:0:2402:0:(client.c:3425:ptlrpc_replay_interpret()) @@@ got reply req@ffff94d408d3c700 x1875450488503040/t103079215111(103079215111) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572512 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572501.632541:0:2402:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:0.0:1788572501.632543:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 103079215110 gen 4 00000100:00100000:0.0:1788572501.632548:0:2402:0:(client.c:3156:ptlrpc_free_committed()) @@@ stopping search req@ffff94d408d3c700 x1875450488503040/t103079215111(103079215111) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572512 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572501.632730:0:2402:0:(recover.c:53:ptlrpc_replay_next()) import ffff94d405bab000 from lustre-MDT0000_UUID committed 103079215110 last 103079215111 00000100:00080000:0.0:1788572501.632742:0:2402:0:(client.c:3540:ptlrpc_replay_req()) @@@ REPLAY req@ffff94d408c55880 x1875450488503552/t103079215112(103079215112) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572474 ref 1 fl New:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572501.632755:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488503040:192.168.202.125@tcp:36:mcreate.0 00000100:00100000:0.0:1788572501.632830:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488503552:192.168.202.125@tcp:36:rm.0 00010000:00010000:3.0:1788572501.637376:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572501.656073:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 103079215111 gen 4 00000100:00100000:0.0:1788572501.656103:0:2402:0:(client.c:3156:ptlrpc_free_committed()) @@@ stopping search req@ffff94d408c55880 x1875450488503552/t103079215112(103079215112) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572517 ref 2 fl Rpc:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572501.656119:0:2402:0:(client.c:3425:ptlrpc_replay_interpret()) @@@ got reply req@ffff94d408c55880 x1875450488503552/t103079215112(103079215112) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572517 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572501.656127:0:2402:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:0.0:1788572501.656130:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 103079215111 gen 4 00000100:00100000:0.0:1788572501.656134:0:2402:0:(client.c:3156:ptlrpc_free_committed()) @@@ stopping search req@ffff94d408c55880 x1875450488503552/t103079215112(103079215112) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572517 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572501.656140:0:2402:0:(recover.c:53:ptlrpc_replay_next()) import ffff94d405bab000 from lustre-MDT0000_UUID committed 103079215111 last 103079215112 00000100:00080000:0.0:1788572501.656147:0:2402:0:(client.c:3540:ptlrpc_replay_req()) @@@ REPLAY req@ffff94d408d3e300 x1875450488505600/t103079215114(103079215114) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 488/456 e 0 to 0 dl 1788572481 ref 1 fl New:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572501.656158:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488503552:192.168.202.125@tcp:36:rm.0 00000100:00100000:0.0:1788572501.656176:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488505600:192.168.202.125@tcp:36:dd.0 00000100:00080000:0.0:1788572501.663255:0:2402:0:(client.c:3425:ptlrpc_replay_interpret()) @@@ got reply req@ffff94d408d3e300 x1875450488505600/t103079215114(103079215114) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 488/456 e 0 to 0 dl 1788572517 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572501.663271:0:2402:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:0.0:1788572501.663273:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 103079215111 gen 4 00000100:00100000:0.0:1788572501.663280:0:2402:0:(client.c:3156:ptlrpc_free_committed()) @@@ stopping search req@ffff94d408c55880 x1875450488503552/t103079215112(103079215112) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572517 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572501.663289:0:2402:0:(recover.c:53:ptlrpc_replay_next()) import ffff94d405bab000 from lustre-MDT0000_UUID committed 103079215111 last 103079215114 00000100:00080000:0.0:1788572501.663297:0:2402:0:(client.c:3540:ptlrpc_replay_req()) @@@ REPLAY req@ffff94d408d3ca80 x1875450488505856/t103079215115(103079215115) o10->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:13/10 lens 440/432 e 0 to 0 dl 1788572481 ref 1 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788572501.663308:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488505600:192.168.202.125@tcp:36:dd.0 00000100:00100000:0.0:1788572501.663330:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488505856:192.168.202.125@tcp:10:dd.0 00000100:00080000:0.0:1788572501.666491:0:2402:0:(client.c:3425:ptlrpc_replay_interpret()) @@@ got reply req@ffff94d408d3ca80 x1875450488505856/t103079215115(103079215115) o10->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:13/10 lens 440/432 e 0 to 0 dl 1788572517 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:0.0:1788572501.666507:0:2402:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:0.0:1788572501.666515:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 103079215111 gen 4 00000100:00100000:0.0:1788572501.666521:0:2402:0:(client.c:3156:ptlrpc_free_committed()) @@@ stopping search req@ffff94d408c55880 x1875450488503552/t103079215112(103079215112) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572517 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572501.666530:0:2402:0:(recover.c:53:ptlrpc_replay_next()) import ffff94d405bab000 from lustre-MDT0000_UUID committed 103079215111 last 103079215115 00000100:00080000:0.0:1788572501.666534:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:0.0:1788572501.667043:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488505856:192.168.202.125@tcp:10:dd.0 00010000:00080000:3.0:1788572501.667100:0:68182:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff94d405cc3800 to lustre-MDT0000_UUID@192.168.202.125@tcp 00010000:00010000:3.0:1788572501.667105:0:68182:0:(ldlm_request.c:2888:ldlm_cancel_unused_locks_for_replay()) Dropping as many unused locks as possible before replay for namespace lustre-MDT0000-mdc-ffff94d405cc3800 (3) 00000080:00200000:3.0:1788572501.667151:0:68182:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572501.667167:0:68182:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:3.0:1788572501.667245:0:68182:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166a00/0x1e260726fbbc0a9e lrc: 0/0,0 mode: --/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0xec901abfeb4e0466 expref: -99 pid: 67901 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572501.667292:0:68182:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0a51 lrc: 0/0,0 mode: --/PR res: [0x20000a811:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0xec901abfeb4e0340 expref: -99 pid: 67224 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572501.667305:0:68182:0:(ldlm_request.c:2901:ldlm_cancel_unused_locks_for_replay()) Canceled 0 unused locks from namespace lustre-MDT0000-mdc-ffff94d405cc3800 00010000:00010000:3.0:1788572501.668045:0:68182:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 3/0,0 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x0 rrc: 2 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0451 expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00000100:00100000:0.0:1788572501.668121:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488511616:192.168.202.125@tcp:101:ldlm_lock_repla.0 00010000:00080000:3.0:1788572501.668367:0:68182:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 1(8) 00010000:00010000:1.0:1788572501.675818:0:2402:0:(ldlm_request.c:2777:replay_lock_interpret()) ### replayed lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 3/0,0 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x0 rrc: 2 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e06dc expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00000100:00080000:1.0:1788572501.675837:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to REPLAY_WAIT 00000100:00100000:1.0:1788572501.675867:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488511616:192.168.202.125@tcp:101:ldlm_lock_repla.0 00000100:00100000:1.0:1788572501.675891:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488511744:192.168.202.125@tcp:400:ptlrpcd_rcv.0 00000100:00100000:1.0:1788572501.688805:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 103079215115 gen 4 00000100:00080000:1.0:1788572501.688847:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from REPLAY_WAIT to RECOVER 00000100:00080000:1.0:1788572501.688863:0:2402:0:(client.c:3234:ptlrpc_resend_req()) @@@ going to resend req@ffff94d4049bca80 x1875450488511744/t0(0) o400->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572517 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572501.688881:0:2402:0:(client.c:3243:ptlrpc_resend_req()) @@@ it has reply, so skip it req@ffff94d4049bca80 x1875450488511744/t0(0) o400->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572517 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572501.688889:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from RECOVER to FULL 00000100:02000000:2.0:1788572501.689299:0:2402:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) lustre-MDT0000-mdc-ffff94d405cc3800: Connection restored to 192.168.202.125@tcp (at 192.168.202.125@tcp) 00000100:00080000:2.0:1788572501.689307:0:2402:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000af1897cb): req@ffff94d408c60380 x1875450488507136/t0(0) o16->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 440/432 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788572501.689331:0:2402:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000af1897cb): req@ffff94d408c63800 x1875450488507008/t0(0) o4->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:13/10 lens 496/448 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788572501.689340:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488511744:192.168.202.125@tcp:400:ptlrpcd_rcv.0 00000008:00010000:2.0:1788572501.699901:0:2405:0:(osc_cache.c:910:osc_extent_finish()) ### extent: 000000008b6de3dd ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 2/0,0 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x0 rrc: 2 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e06dc expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00000008:00010000:2.0:1788572501.699949:0:2405:0:(osc_cache.c:496:__osc_extent_remove()) ### extent: 000000008b6de3dd ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 2/0,0 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x0 rrc: 2 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e06dc expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00000100:00100000:2.0:1788572501.699984:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:0:1875450488507008:192.168.202.125@tcp:4:dd.0 00000100:00100000:3.0:1788572501.706377:0:2405:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 107374182401 gen 4 00000100:00100000:3.0:1788572501.706426:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:0:1875450488507136:192.168.202.125@tcp:16:kworker.0 00000080:00200000:2.0:1788572501.707261:0:470:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1788572502.213459:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488511872:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572502.220007:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788572502.220031:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488512000:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572502.228409:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488512000:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572502.228441:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488511872:192.168.202.125@tcp:400:kworker.0 00010000:00010000:1.0:1788572502.662407:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572503.685834:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572504.713411:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:0.0:1788572505.240744:0:68225:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788572505.240757:0:68225:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:0.0:1788572505.240780:0:68225:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572505.240838:0:68225:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 68225 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572505.240848:0:68225:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 68225 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572505.240858:0:68225:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 68225 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572505.240882:0:68225:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:68225:1875450488512128:192.168.202.125@tcp:101:lfs.0 00000100:00100000:0.0:1788572505.240964:0:68225:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4054116c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572505.253148:0:68225:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:68225:1875450488512128:192.168.202.125@tcp:101:lfs.0 00010000:00010000:0.0:1788572505.253176:0:68225:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572505.253189:0:68225:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572505.253218:0:68225:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572505.253230:0:68225:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572505.253242:0:68225:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572505.253251:0:68225:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572505.253259:0:68225:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572505.253269:0:68225:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408d3c700 x1875450488512128/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572521 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572505.253280:0:68225:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572505.253289:0:68225:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788572505.253295:0:68225:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788572505.253330:0:68225:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1788572505.253334:0:68225:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0ac1 00000080:00010000:0.0:1788572505.253338:0:68225:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572505.253341:0:68225:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0ac1 from it ffffb68a423dfcb8 00010000:00010000:0.0:1788572505.253343:0:68225:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572505.253349:0:68225:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1788572505.253902:0:68225:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1788572505.253907:0:68225:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:0.0:1788572505.253920:0:68225:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572505.253928:0:68225:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572505.253936:0:68225:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572505.253940:0:68225:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572505.253944:0:68225:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572505.253946:0:68225:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0ac1 00000080:00010000:0.0:1788572505.253949:0:68225:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0ac1 from it ffffb68a423dfcb8 00010000:00010000:0.0:1788572505.253951:0:68225:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572505.253956:0:68225:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788572505.254036:0:68225:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:0.0:1788572505.254045:0:68225:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572505.254052:0:68225:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572505.254061:0:68225:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788572505.254067:0:68225:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572505.254072:0:68225:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572505.254074:0:68225:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0ac1 00000080:00010000:0.0:1788572505.254077:0:68225:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0ac1 from it ffffb68a423dfbb0 00010000:00010000:0.0:1788572505.254080:0:68225:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572505.254087:0:68225:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788572505.254095:0:68225:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 44 00000080:00200008:0.0:1788572505.254102:0:68225:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 100000 00000080:00200080:0.0:1788572505.254192:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:0.0:1788572505.254239:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084de800 uarg=0000000000000000 00800000:00000004:0.0:1788572505.254246:0:68225:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0000-mdc-ffff94d405cc3800: age 3611, max_age 3657 00000100:00100000:0.0:1788572505.254277:0:68225:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:68225:1875450488512256:192.168.202.125@tcp:41:lfs.0 00000100:00100000:0.0:1788572505.254344:0:68225:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4054116c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572505.258482:0:68225:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:68225:1875450488512256:192.168.202.125@tcp:41:lfs.0 00800000:00000004:0.0:1788572505.258511:0:68225:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0000-mdc-ffff94d405cc3800: update ffff94d412909e10 cache blocks 317130/353529 objects 1023603/1024000 00000080:00200080:0.0:1788572505.258587:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:0.0:1788572505.258605:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084de800 uarg=0000000000000000 00800000:00000004:0.0:1788572505.258612:0:68225:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0001-mdc-ffff94d405cc3800: age 3611, max_age 3657 00000100:00100000:0.0:1788572505.258647:0:68225:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:68225:1875450488512384:192.168.202.125@tcp:41:lfs.0 00000100:00100000:0.0:1788572505.258725:0:68225:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4054116c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572505.269492:0:68225:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:68225:1875450488512384:192.168.202.125@tcp:41:lfs.0 00800000:00000004:2.0:1788572505.269525:0:68225:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0001-mdc-ffff94d405cc3800: update ffff94d407db9e10 cache blocks 317207/353529 objects 1023684/1024000 00000080:00200080:2.0:1788572505.269622:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269639:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269647:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269651:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269655:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269659:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269663:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269666:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269670:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269673:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269677:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269680:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269684:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269687:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269691:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269694:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269698:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269701:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269705:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269709:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269713:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269716:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269720:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269723:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269727:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269731:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269735:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269739:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269743:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269746:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269750:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269753:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269757:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269760:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269764:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269768:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269772:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269775:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269779:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269783:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269786:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269790:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269794:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269798:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269802:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269805:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269809:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269813:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269816:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269820:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269824:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269827:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269831:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269835:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269839:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269843:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269846:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269850:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269855:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269858:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269861:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269865:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269869:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269873:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269877:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269881:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269884:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269887:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269891:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269894:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269898:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269901:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269904:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269907:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269911:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269914:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269918:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269922:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269936:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269940:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269944:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269947:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269951:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269954:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269958:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269961:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269965:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269969:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269973:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269977:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269981:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269985:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269989:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.269993:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.269997:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.270000:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.270004:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.270009:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.270014:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.270017:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.270021:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.270025:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.270028:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.270032:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.270036:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.270039:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.270042:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.270046:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.270050:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.270054:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.270057:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.270061:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.270065:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.270069:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.270072:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.270075:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.270079:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.270082:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.270087:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.270091:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.270094:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.270098:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.270102:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.270106:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.270109:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00800000:00000080:2.0:1788572505.270113:0:68225:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.270117:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.270123:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00020000:00000004:2.0:1788572505.270130:0:68225:0:(obd_class.h:1091:obd_statfs()) lustre-OST0000-osc-ffff94d405cc3800: age 3611, max_age 3657 00000100:00080000:2.0:1788572505.270144:0:68225:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3591 00000100:00080000:2.0:1788572505.270553:0:68225:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d405cc3800: import ffff94d405bad000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572505.270582:0:68225:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8437760, lost: 0. 00000100:00100000:2.0:1788572505.270612:0:68225:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488512512/t0(0) o8->lustre-OST0000-osc-ffff94d405cc3800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572505.270659:0:68225:0:(client.c:1832:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff94d408c60700 x1875450488512640/t0(0) o13->lustre-OST0000-osc-ffff94d405cc3800@192.168.202.125@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572505.270669:0:68225:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d412842d80 going to sleep for 0 seconds 00000100:00100000:1.0:1788572505.270733:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488512512:192.168.202.125@tcp:8:lfs.0 00000100:00100000:1.0:1788572505.274340:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-OST0000-osc-ffff94d405cc3800: committing for last_committed 4294968859 gen 16 00000100:00080000:1.0:1788572505.274359:0:2402:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff94d405cc3800: connect to target with instance 1 00000100:00080000:1.0:1788572505.274370:0:2402:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff94d405cc3800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:1.0:1788572505.274385:0:2402:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:1.0:1788572505.274389:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bad000 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00000080:00000004:1.0:1788572505.274398:0:2402:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:1.0:1788572505.274403:0:2402:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:1.0:1788572505.274419:0:2402:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 0000000064e702af): req@ffff94d408c60700 x1875450488512640/t0(0) o13->lustre-OST0000-osc-ffff94d405cc3800@192.168.202.125@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572505.274454:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488512512:192.168.202.125@tcp:8:lfs.0 00000100:00100000:2.0:1788572505.274587:0:68225:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d412842d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572505.277411:0:68225:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:68225:1875450488512640:192.168.202.125@tcp:13:lfs.0 00020000:00000004:2.0:1788572505.277445:0:68225:0:(obd_class.h:1117:obd_statfs()) lustre-OST0000-osc-ffff94d405cc3800: update ffff94d407dbd190 cache blocks 901369/958279 objects 261733/262144 00000080:00200080:2.0:1788572505.277535:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.277554:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00020000:00000004:2.0:1788572505.277563:0:68225:0:(obd_class.h:1091:obd_statfs()) lustre-OST0001-osc-ffff94d405cc3800: age 3611, max_age 3657 00000100:00080000:2.0:1788572505.277602:0:68225:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3591 00000100:00080000:2.0:1788572505.277667:0:68225:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d405cc3800: import ffff94d405bab800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572505.277676:0:68225:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8437760, lost: 0. 00000100:00100000:2.0:1788572505.277700:0:68225:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488512768/t0(0) o8->lustre-OST0001-osc-ffff94d405cc3800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572505.277734:0:68225:0:(client.c:1832:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff94d408c60700 x1875450488512896/t0(0) o13->lustre-OST0001-osc-ffff94d405cc3800@192.168.202.125@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572505.277755:0:68225:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d412842d80 going to sleep for 0 seconds 00000100:00100000:0.0:1788572505.277799:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488512768:192.168.202.125@tcp:8:lfs.0 00000100:00100000:0.0:1788572505.281642:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-OST0001-osc-ffff94d405cc3800: committing for last_committed 0 gen 17 00000100:00080000:0.0:1788572505.281662:0:2402:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff94d405cc3800: connect to target with instance 1 00000100:00080000:0.0:1788572505.281673:0:2402:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff94d405cc3800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:0.0:1788572505.281687:0:2402:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:0.0:1788572505.281692:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab800 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00000080:00000004:0.0:1788572505.281703:0:2402:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:0.0:1788572505.281708:0:2402:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:0.0:1788572505.281724:0:2402:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 0000000064e702af): req@ffff94d408c60700 x1875450488512896/t0(0) o13->lustre-OST0001-osc-ffff94d405cc3800@192.168.202.125@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572505.281753:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488512768:192.168.202.125@tcp:8:lfs.0 00000100:00100000:2.0:1788572505.282296:0:68225:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d412842d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572505.284681:0:68225:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:68225:1875450488512896:192.168.202.125@tcp:13:lfs.0 00020000:00000004:2.0:1788572505.284709:0:68225:0:(obd_class.h:1117:obd_statfs()) lustre-OST0001-osc-ffff94d405cc3800: update ffff94d407dbb7d0 cache blocks 901369/958279 objects 261733/262144 00000080:00200080:2.0:1788572505.284789:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.284807:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.284814:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.284819:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.284823:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.284827:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.284831:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.284834:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.284839:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.284843:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.284847:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.284851:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.284855:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.284859:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.284864:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.284867:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.284871:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.284875:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.284878:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.284882:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.284886:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.284889:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.284893:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.284897:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.284901:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.284905:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.284909:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.284913:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.284917:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.284921:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285114:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285120:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285124:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285127:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285131:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285134:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285138:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285141:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285145:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285159:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285164:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285169:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285173:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285177:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285181:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285185:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285189:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285192:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285196:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285235:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285242:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285247:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285251:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285254:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285257:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285261:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285265:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285268:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285272:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285275:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285279:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285282:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285287:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285291:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285295:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285298:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285301:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285305:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285309:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285312:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285316:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285319:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285323:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285327:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285331:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285335:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285339:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285342:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285346:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285350:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285353:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285357:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285360:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285364:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285368:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285372:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285375:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285379:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285383:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285387:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285392:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285395:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285399:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285403:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285408:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285410:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285414:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.285419:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.285422:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.287275:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.287384:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.287395:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.287401:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.287404:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.287408:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.287412:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.287415:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.287419:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.287422:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.287425:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.287440:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.287444:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.287448:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.287451:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.287455:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.287458:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.287463:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.287466:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.287479:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.287484:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.287489:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.287492:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.287496:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.287500:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200080:2.0:1788572505.287504:0:68225:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffe132dbdb0 00020000:00000080:2.0:1788572505.287507:0:68225:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541dc00 uarg=0000000000000000 00000080:00200008:2.0:1788572505.287523:0:68225:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 100000 00010000:00010000:2.0:1788572505.739666:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572506.759384:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572507.340228:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488513024:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572507.340603:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788572507.340612:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488513152:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572507.347840:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488513408:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572507.347942:0:2405:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:2.0:1788572507.347950:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488513280:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572507.347975:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488513536:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572507.360662:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488513280:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572507.360683:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488513536:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572507.360706:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488513408:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572507.361009:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488513152:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572507.374200:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488513024:192.168.202.125@tcp:400:kworker.0 00010000:00010000:3.0:1788572507.783083:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572508.805428:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572509.832181:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572510.853572:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572511.877604:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572512.461133:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488513664:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572512.461252:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488513920:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572512.461281:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488514176:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572512.461313:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:1.0:1788572512.461320:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488513792:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572512.461343:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488514048:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572512.470241:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488513792:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572512.470261:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488514048:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572512.470271:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488513664:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572512.470285:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488513920:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572512.470297:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488514176:192.168.202.125@tcp:400:kworker.0 00000001:02000400:2.0:1788572512.558045:0:68507:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 00010000:00010000:1.0:1788572512.902418:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572513.936738:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788572514.577911:0:68655:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 00010000:00010000:1.0:1788572514.949490:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572515.974518:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:0.0:1788572516.590873:0:68694:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788572516.590887:0:68694:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:0.0:1788572516.590910:0:68694:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572516.590927:0:68694:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572516.590943:0:68694:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788572516.590952:0:68694:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572516.590961:0:68694:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572516.590965:0:68694:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0ac1 00000080:00010000:0.0:1788572516.590972:0:68694:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0ac1 from it ffffb68a40dabcb8 00010000:00010000:0.0:1788572516.590976:0:68694:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572516.590985:0:68694:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:0.0:1788572516.591990:0:68694:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1788572516.591997:0:68694:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:0.0:1788572516.592014:0:68694:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572516.592025:0:68694:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788572516.592035:0:68694:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788572516.592042:0:68694:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572516.592047:0:68694:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572516.592050:0:68694:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0ac1 00000080:00010000:0.0:1788572516.592056:0:68694:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0ac1 from it ffffb68a40dabcb8 00010000:00010000:0.0:1788572516.592059:0:68694:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788572516.592071:0:68694:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1788572516.592177:0:68694:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:0.0:1788572516.592186:0:68694:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572516.592193:0:68694:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1788572516.592350:0:68694:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1788572516.592358:0:68694:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572516.592363:0:68694:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572516.592365:0:68694:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0ac1 00000080:00010000:0.0:1788572516.592369:0:68694:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0ac1 from it ffffb68a40dabbb0 00010000:00010000:0.0:1788572516.592372:0:68694:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1788572516.592380:0:68694:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:0.0:1788572516.592388:0:68694:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 44 00000080:00200008:0.0:1788572516.592395:0:68694:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 100000 00000080:00200080:0.0:1788572516.592580:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.592594:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00800000:00000004:0.0:1788572516.592601:0:68694:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0000-mdc-ffff94d405cc3800: age 3658, max_age 3669 00000100:00100000:0.0:1788572516.592644:0:68694:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:68694:1875450488514304:192.168.202.125@tcp:41:lfs.0 00000100:00100000:0.0:1788572516.592730:0:68694:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d407e139c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572516.601478:0:68694:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:68694:1875450488514304:192.168.202.125@tcp:41:lfs.0 00800000:00000004:0.0:1788572516.601508:0:68694:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0000-mdc-ffff94d405cc3800: update ffff94d412909e10 cache blocks 317130/353529 objects 1023603/1024000 00000080:00200080:0.0:1788572516.601601:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.601622:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00800000:00000004:0.0:1788572516.601632:0:68694:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0001-mdc-ffff94d405cc3800: age 3658, max_age 3669 00000100:00100000:0.0:1788572516.601672:0:68694:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:68694:1875450488514432:192.168.202.125@tcp:41:lfs.0 00000100:00100000:0.0:1788572516.601748:0:68694:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d407e139c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572516.605193:0:68694:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:68694:1875450488514432:192.168.202.125@tcp:41:lfs.0 00800000:00000004:0.0:1788572516.605264:0:68694:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0001-mdc-ffff94d405cc3800: update ffff94d407db9e10 cache blocks 317207/353529 objects 1023684/1024000 00000080:00200080:0.0:1788572516.605330:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605348:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605356:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605361:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605366:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605370:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605374:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605378:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605382:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605385:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605390:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605393:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605396:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605400:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605403:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605406:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605410:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605413:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605417:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605420:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605423:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605427:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605430:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605434:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605437:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605440:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605444:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605447:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605451:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605454:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605457:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605460:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605464:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605466:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605469:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605472:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605475:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605478:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605483:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605486:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605490:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605492:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605496:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605499:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605503:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605506:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605509:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605513:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605516:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605519:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605532:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605537:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605542:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605546:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605549:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605553:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605557:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605560:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605564:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605568:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605572:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605575:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605579:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605583:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605587:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605590:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605594:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605598:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605601:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605605:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605608:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605611:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605615:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605618:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605622:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605625:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605628:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605631:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605635:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605638:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605642:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605645:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605649:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605652:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605656:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605659:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605662:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605666:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605671:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605674:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605677:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605681:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605684:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605688:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605691:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605695:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605699:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605702:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605706:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605710:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605713:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605716:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605720:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605724:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605728:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605732:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605737:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605740:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605744:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605747:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605752:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605755:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605759:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605763:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605767:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605771:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605775:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605779:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605782:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605786:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605789:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605792:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605796:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605800:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605803:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00800000:00000080:0.0:1788572516.605807:0:68694:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.605880:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.605890:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00020000:00000004:0.0:1788572516.605899:0:68694:0:(obd_class.h:1091:obd_statfs()) lustre-OST0000-osc-ffff94d405cc3800: age 3658, max_age 3669 00000100:00100000:0.0:1788572516.605936:0:68694:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:68694:1875450488514560:192.168.202.125@tcp:13:lfs.0 00000100:00100000:0.0:1788572516.606014:0:68694:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d407e139c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572516.611934:0:68694:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:68694:1875450488514560:192.168.202.125@tcp:13:lfs.0 00020000:00000004:0.0:1788572516.611963:0:68694:0:(obd_class.h:1117:obd_statfs()) lustre-OST0000-osc-ffff94d405cc3800: update ffff94d407dbd190 cache blocks 901369/958279 objects 261733/262144 00000080:00200080:0.0:1788572516.612073:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.612092:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00020000:00000004:0.0:1788572516.612100:0:68694:0:(obd_class.h:1091:obd_statfs()) lustre-OST0001-osc-ffff94d405cc3800: age 3658, max_age 3669 00000100:00100000:0.0:1788572516.612137:0:68694:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:68694:1875450488514688:192.168.202.125@tcp:13:lfs.0 00000100:00100000:0.0:1788572516.612234:0:68694:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d407e139c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572516.619441:0:68694:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:68694:1875450488514688:192.168.202.125@tcp:13:lfs.0 00020000:00000004:0.0:1788572516.619477:0:68694:0:(obd_class.h:1117:obd_statfs()) lustre-OST0001-osc-ffff94d405cc3800: update ffff94d407dbb7d0 cache blocks 901369/958279 objects 261733/262144 00000080:00200080:0.0:1788572516.619628:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619649:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619655:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619660:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619664:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619668:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619673:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619676:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619681:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619684:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619689:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619692:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619697:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619700:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619704:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619711:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619714:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619718:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619722:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619725:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619729:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619734:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619738:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619741:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619744:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619748:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619751:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619754:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619758:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619761:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619765:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619768:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619773:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619776:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619779:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619782:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619786:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619788:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619792:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619794:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619798:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619802:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619805:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619808:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619824:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619830:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619835:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619839:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619843:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619846:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619850:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619854:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619858:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619861:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619865:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619869:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619873:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619877:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619880:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619884:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619888:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619891:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619896:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619899:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619904:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619908:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619911:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619914:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619918:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619922:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619925:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619930:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619933:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619937:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619941:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619944:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619948:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619952:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619955:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619958:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619962:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619965:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619969:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619972:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619975:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619978:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619982:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619985:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619989:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.619992:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.619996:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.620000:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.620004:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.620007:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.620011:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.620015:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.620019:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.620022:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.620026:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.620029:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.620033:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.620037:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.620041:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.620045:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.620049:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.620053:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.620057:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.620060:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.620064:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.620067:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.620071:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.620075:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.620079:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.620082:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.620086:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.620089:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.620093:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.620097:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.620101:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.620105:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.620108:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.620112:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.620116:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.620119:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200080:0.0:1788572516.620123:0:68694:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc07b95660 00020000:00000080:0.0:1788572516.620126:0:68694:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dcc00 uarg=0000000000000000 00000080:00200008:0.0:1788572516.620140:0:68694:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 100000 00000080:00200000:1.0:1788572516.678797:0:68711:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572516.678831:0:68711:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572516.678852:0:68711:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572516.678869:0:68711:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1788572516.678877:0:68711:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572516.678885:0:68711:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572516.678889:0:68711:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0ac1 00000080:00010000:1.0:1788572516.678895:0:68711:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0ac1 from it ffffb68a41067bf8 00010000:00010000:1.0:1788572516.678899:0:68711:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572516.678908:0:68711:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1788572516.678924:0:68711:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572516.678933:0:68711:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f131a.replay-single, flags=265 00000080:00200000:1.0:1788572516.678939:0:68711:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x20000a811:0x6:0x0](ffff94d4034963c8), inode mode 81a4 mask 44 00000080:00200008:1.0:1788572516.678945:0:68711:0:(file.c:1090:ll_file_open()) START file f131a.replay-single:[0x20000a811:0x6:0x0](ffff94d4034963c8), flags 100000 00000002:00010000:1.0:1788572516.678963:0:68711:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x20000a811:0x6:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0240100001 00010000:00010000:1.0:1788572516.679019:0:68711:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0ac8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 68711 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.679027:0:68711:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0ac8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 68711 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572516.679041:0:68711:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572516.679053:0:68711:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d4049bed80 x1875450488514944/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572516.679063:0:68711:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0ac8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 68711 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572516.679085:0:68711:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job cat:9bf287c0-f582-4b83-84c1-15e766218620:68711:1875450488514944:192.168.202.125@tcp:101:cat.0 00000100:00100000:1.0:1788572516.679286:0:68711:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69e40 going to sleep for 11 seconds 00000100:00100000:1.0:1788572516.701929:0:68711:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job cat:9bf287c0-f582-4b83-84c1-15e766218620:68711:1875450488514944:192.168.202.125@tcp:101:cat.0 00010000:00010000:1.0:1788572516.701958:0:68711:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0ac8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 68711 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.701972:0:68711:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0ac8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 68711 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.701980:0:68711:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0ac8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 68711 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.701988:0:68711:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0ac8 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 68711 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.701997:0:68711:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0ac8 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 68711 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.702003:0:68711:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412583600 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:1.0:1788572516.702008:0:68711:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0ac8 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 68711 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.702015:0:68711:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0ac8 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 68711 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.702022:0:68711:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0ac8 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 68711 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.702029:0:68711:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572516.702036:0:68711:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0ac8 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 68711 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.702042:0:68711:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583600/0x1e260726fbbc0ac8 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 68711 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572516.702061:0:68711:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff94d4049bed80 x1875450488514944/t107374182402(107374182402) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/824 e 0 to 0 dl 1788572532 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'cat.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572516.702074:0:68711:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d4049bed80 x1875450488514944/t107374182402(107374182402) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/824 e 0 to 0 dl 1788572532 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'cat.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572516.702092:0:68711:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000002:00100000:1.0:1788572516.702226:0:68711:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d4049bed80 x1875450488514944/t107374182402(107374182402) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 736/824 e 0 to 0 dl 1788572532 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'cat.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1788572516.702309:0:68711:0:(file.c:6285:ll_getattr_dentry()) START file f131a.replay-single:[0x20000a811:0x6:0x0](ffff94d4034963c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572516.702317:0:68711:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x20000a811:0x6:0x0](ffff94d4034963c8),name=f131a.replay-single 00000002:00010000:1.0:1788572516.702337:0:68711:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x20000a811:0x6:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572516.702349:0:68711:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 30 res 8589977617/6 (0 0) 00010000:00010000:1.0:1788572516.702390:0:68711:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc0acf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 68711 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.702398:0:68711:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc0acf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 68711 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.702407:0:68711:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc0acf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 68711 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572516.702427:0:68711:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job cat:9bf287c0-f582-4b83-84c1-15e766218620:68711:1875450488515072:192.168.202.125@tcp:101:cat.0 00000100:00100000:1.0:1788572516.702502:0:68711:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69e40 going to sleep for 11 seconds 00000100:00100000:3.0:1788572516.705275:0:2397:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.125@tcp (source: 12345-192.168.202.125@tcp) 00000100:00100000:3.0:1788572516.705313:0:42439:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875450511509120 00000100:00100000:3.0:1788572516.705324:0:42439:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875450511509120 00000100:00100000:3.0:1788572516.705379:0:42439:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.125@tcp, seq: 93 00000100:00100000:3.0:1788572516.705391:0:42439:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff94d408c57b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:9bf287c0-f582-4b83-84c1-15e766218620+5:80444:x1875450511509120:12345-192.168.202.125@tcp:104: 00010000:00010000:3.0:1788572516.705410:0:42439:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 2/0,0 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x0 rrc: 2 type: IBT gid 0 flags: 0x800420000000000 nid: local remote: 0xec901abfeb4e06dc expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788572516.705492:0:42439:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788572516.705501:0:42439:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff94d408c57b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:9bf287c0-f582-4b83-84c1-15e766218620+5:80444:x1875450511509120:12345-192.168.202.125@tcp:104: Request processed in 112us (236us total) trans 0 rc 0/0 00000100:00100000:3.0:1788572516.705509:0:42439:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.125@tcp, seq: 93 00010000:00010000:1.0:1788572516.705817:0:42449:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788572516.705826:0:42449:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 2/0,0 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x0 rrc: 2 type: IBT gid 0 flags: 0x800420000000000 nid: local remote: 0xec901abfeb4e06dc expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572516.705835:0:42449:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412166e00 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:1.0:1788572516.705840:0:42449:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 2/0,0 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x59 rrc: 2 type: IBT gid 0 flags: 0x800420400000000 nid: local remote: 0xec901abfeb4e06dc expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572516.705849:0:42449:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 2/0,0 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x59 rrc: 2 type: IBT gid 0 flags: 0x800420400000000 nid: local remote: 0xec901abfeb4e06dc expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572516.705859:0:42449:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 3/0,0 mode: PW/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x59 rrc: 2 type: IBT gid 0 flags: 0x800428400000000 nid: local remote: 0xec901abfeb4e06dc expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1788572516.705925:0:42449:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788572516.705984:0:42449:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572516.706020:0:42449:0:(vvp_object.c:117:vvp_conf_set()) [0x20000a811:0x6:0x0]: losing layout lock 00010000:00010000:1.0:1788572516.706087:0:42449:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 2/0,0 mode: --/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x59 rrc: 2 type: IBT gid 0 flags: 0x804c29400000020 nid: local remote: 0xec901abfeb4e06dc expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572516.706095:0:42449:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572516.706130:0:42449:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 1/0,0 mode: --/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x59 rrc: 2 type: IBT gid 0 flags: 0x804c29400000020 nid: local remote: 0xec901abfeb4e06dc expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572516.706139:0:42449:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166e00/0x1e260726fbbc0a97 lrc: 0/0,0 mode: --/PW res: [0x20000a811:0x6:0x0].0x0 bits 0x59/0x59 rrc: 2 type: IBT gid 0 flags: 0x804c29400000020 nid: local remote: 0xec901abfeb4e06dc expref: -99 pid: 67901 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572516.706158:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572516.706160:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572516.706162:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572516.706178:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488515200:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572516.712240:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488515200:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572516.712368:0:68711:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job cat:9bf287c0-f582-4b83-84c1-15e766218620:68711:1875450488515072:192.168.202.125@tcp:101:cat.0 00010000:00010000:1.0:1788572516.712407:0:68711:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc0acf lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08db expref: -99 pid: 68711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.712433:0:68711:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc0acf lrc: 4/1,0 mode: --/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08db expref: -99 pid: 68711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.712441:0:68711:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc0acf lrc: 4/1,0 mode: --/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08db expref: -99 pid: 68711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.712453:0:68711:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc0acf lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e08db expref: -99 pid: 68711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.712462:0:68711:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc0acf lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08db expref: -99 pid: 68711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.712476:0:68711:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc0acf lrc: 5/2,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e08db expref: -99 pid: 68711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.712484:0:68711:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc0acf lrc: 5/2,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e08db expref: -99 pid: 68711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.712492:0:68711:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc0acf lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e08db expref: -99 pid: 68711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572516.712503:0:68711:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d4049bfb80 x1875450488515072/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/800 e 0 to 0 dl 1788572532 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572516.712516:0:68711:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc0acf lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08db expref: -99 pid: 68711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.712537:0:68711:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589977617/6 (0 0) 00000080:00200000:1.0:1788572516.712545:0:68711:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1788572516.712621:0:68711:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a811:0x6:0x0](ffff94d4034963c8) for lock 0x1e260726fbbc0acf 00000080:00010000:1.0:1788572516.712627:0:68711:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a811:0x6:0x0](ffff94d4034963c8) 00000080:00010000:1.0:1788572516.712630:0:68711:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0acf from it ffffb68a41067cd8 00010000:00010000:1.0:1788572516.712634:0:68711:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc0acf lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e08db expref: -99 pid: 68711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.712642:0:68711:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc0acf lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08db expref: -99 pid: 68711 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572516.712680:0:68711:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x6:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00010000:00010000:1.0:1788572516.712696:0:68711:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 31 res 8589977617/6 (0 0) 00000080:00010000:1.0:1788572516.712703:0:68711:0:(file.c:7048:ll_layout_intent()) ### lustre: requeue layout lock for file [0x20000a811:0x6:0x0](ffff94d4034963c8) 00000002:00010000:1.0:1788572516.712713:0:68711:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x20000a811:0x6:0x0]) in obj [0x20000a811:0x6:0x0], intent: layout flags 0 00010000:00010000:1.0:1788572516.712748:0:68711:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0ad6 lrc: 3/1,0 mode: --/CR res: [0x20000a811:0x6:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 68711 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.712754:0:68711:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0ad6 lrc: 3/1,0 mode: --/CR res: [0x20000a811:0x6:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT flags: 0x0 pid: 68711 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.712763:0:68711:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0ad6 lrc: 3/1,0 mode: --/CR res: [0x20000a811:0x6:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT flags: 0x0 pid: 68711 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572516.712784:0:68711:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job cat:9bf287c0-f582-4b83-84c1-15e766218620:68711:1875450488515328:192.168.202.125@tcp:101:cat.0 00000100:00100000:1.0:1788572516.712866:0:68711:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411b69e40 going to sleep for 11 seconds 00000100:00100000:0.0:1788572516.716974:0:68711:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job cat:9bf287c0-f582-4b83-84c1-15e766218620:68711:1875450488515328:192.168.202.125@tcp:101:cat.0 00010000:00010000:0.0:1788572516.717010:0:68711:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0ad6 lrc: 4/1,0 mode: CR/CR res: [0x20000a811:0x6:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e08e2 expref: -99 pid: 68711 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572516.717025:0:68711:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0ad6 lrc: 4/1,0 mode: CR/CR res: [0x20000a811:0x6:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08e2 expref: -99 pid: 68711 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572516.717041:0:68711:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff94d4049bfb80 x1875450488515328/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 376/560 e 0 to 0 dl 1788572532 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572516.717052:0:68711:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 216 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0ad6 lrc: 3/1,0 mode: CR/CR res: [0x20000a811:0x6:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08e2 expref: -99 pid: 68711 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788572516.717064:0:68711:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0ad6 lrc: 3/1,0 mode: CR/CR res: [0x20000a811:0x6:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08e2 expref: -99 pid: 68711 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572516.717076:0:68711:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 16 res 8589977617/6 (0 0) 00000080:00200000:0.0:1788572516.717082:0:68711:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x20000a811:0x6:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1788572516.717099:0:68711:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a811:0x6:0x0](ffff94d4034963c8) for lock 0x1e260726fbbc0ad6 00000080:00010000:0.0:1788572516.717120:0:68711:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0ad6 from it ffffb68a41067b68 00010000:00010000:0.0:1788572516.717125:0:68711:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0ad6 lrc: 3/1,0 mode: CR/CR res: [0x20000a811:0x6:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e08e2 expref: -99 pid: 68711 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572516.717136:0:68711:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0ad6 lrc: 2/0,0 mode: CR/CR res: [0x20000a811:0x6:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08e2 expref: -99 pid: 68711 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572516.717160:0:68711:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0ad6 lrc: 2/1,0 mode: CR/CR res: [0x20000a811:0x6:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08e2 expref: -99 pid: 68711 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572516.717171:0:68711:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0ad6 lrc: 2/1,0 mode: CR/CR res: [0x20000a811:0x6:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08e2 expref: -99 pid: 68711 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788572516.717189:0:68711:0:(file.c:6931:ll_layout_lock_set()) ### file [0x20000a811:0x6:0x0](00000000016d5b0a) being reconfigured ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0ad6 lrc: 3/1,0 mode: CR/CR res: [0x20000a811:0x6:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08e2 expref: -99 pid: 68711 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572516.717275:0:68711:0:(file.c:6826:ll_layout_conf()) [0x20000a811:0x6:0x0]: layout version change: 4294967294 -> 2 00010000:00010000:0.0:1788572516.717296:0:68711:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0ad6 lrc: 3/1,0 mode: CR/CR res: [0x20000a811:0x6:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4e08e2 expref: -99 pid: 68711 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572516.717304:0:68711:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0ad6 lrc: 2/0,0 mode: CR/CR res: [0x20000a811:0x6:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e08e2 expref: -99 pid: 68711 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000001:00010000:0.0:1788572516.717320:0:68711:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000a811:0x6:0x0] 00000002:00010000:0.0:1788572516.717343:0:68711:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000002:00010000:0.0:1788572516.717349:0:68711:0:(mdc_dev.c:1007:mdc_lock_init()) ### lock ffff94d41257d330, mdc lock ffff94d41242a4c0, flags 400001002 00010000:00010000:0.0:1788572516.717359:0:68711:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 6 res 8589977617/6 (0 0) 00010000:00010000:0.0:1788572516.717394:0:68711:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 3/1,0 mode: --/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 68711 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572516.717400:0:68711:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 3/1,0 mode: --/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT flags: 0x0 pid: 68711 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1788572516.717433:0:68711:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:0.0:1788572516.718750:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1788572516.718778:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488515456:192.168.202.125@tcp:101:cat.0 00010000:00010000:0.0:1788572516.733960:0:2403:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 6/2,0 mode: --/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 68711 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572516.733987:0:2403:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 6/2,0 mode: --/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08e9 expref: -99 pid: 68711 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572516.734002:0:2403:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 6/2,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e08e9 expref: -99 pid: 68711 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572516.734012:0:2403:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 6/2,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08e9 expref: -99 pid: 68711 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788572516.734027:0:2403:0:(mdc_dev.c:456:mdc_lock_lvb_update()) ### acquired size 8, setting rss=8; kms=8, end=0 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 5/2,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4e08e9 expref: -99 pid: 68711 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1788572516.734037:0:2403:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1788572516.734056:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 6/2,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e08e9 expref: -99 pid: 68711 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572516.734066:0:2403:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 5/1,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e08e9 expref: -99 pid: 68711 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572516.734072:0:2403:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 5/1,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e08e9 expref: -99 pid: 68711 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572516.734084:0:2403:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 4/0,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e08e9 expref: -99 pid: 68711 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1788572516.734104:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488515456:192.168.202.125@tcp:101:cat.0 00000080:00200000:2.0:1788572516.734199:0:68711:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000a811:0x6:0x0] updating i_size 8 i_blocks 8 00000020:00010000:2.0:1788572516.734235:0:68711:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff94d412aa1720 (ffff94d414dda488/0) at cl_lock_release():245 00000020:00010000:2.0:1788572516.734239:0:68711:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d412aa1720 (ffff94d414dda488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572516.734244:0:68711:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d41257d330 (ffff94d405fa02d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572516.734249:0:68711:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d412aa1720 (ffff94d414dda488/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572516.734252:0:68711:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d41257d330 (ffff94d414dda488/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572516.734293:0:68711:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x6:0x0] ignore/verify layout 0/1, layout version 2 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788572516.734399:0:68711:0:(file.c:2483:do_file_read_iter()) START file f131a.replay-single:[0x20000a811:0x6:0x0], ppos: 0, count: 4194304 00000080:00200000:2.0:1788572516.734419:0:68711:0:(rw.c:1925:ll_readpage()) fast read pgno: 0 00000080:00200000:2.0:1788572516.734461:0:68711:0:(file.c:2090:ll_file_io_generic()) f131a.replay-single: read ppos: 0, bytes: 4194304 00000080:00200000:2.0:1788572516.734479:0:68711:0:(file.c:1848:ll_io_set_mirror()) f131a.replay-single: desiginated mirror: 0 00000020:00200000:2.0:1788572516.734486:0:68711:0:(cl_io.c:214:cl_io_rw_init()) header@00000000386dd75d[0x110, 1, [0x20000a811:0x6:0x0] hash] 00000020:00200000:2.0:1788572516.734488:0:68711:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 4194304) 0 0 00000080:00200000:2.0:1788572516.734492:0:68711:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572516.734508:0:68711:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1048576) 4194304, 1048576 00020000:00200000:2.0:1788572516.734512:0:68711:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572516.734550:0:68711:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 255] 00000002:00010000:2.0:1788572516.734564:0:68711:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000002:00010000:2.0:1788572516.734570:0:68711:0:(mdc_dev.c:1007:mdc_lock_init()) ### lock ffff94d4057bd7b0, mdc lock ffff94d40857a390, flags 0 00010000:00010000:2.0:1788572516.734585:0:68711:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 2/1,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 5 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e08e9 expref: -99 pid: 68711 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572516.734597:0:68711:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 2/1,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e08e9 expref: -99 pid: 68711 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572516.734606:0:68711:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 6/2,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e08e9 expref: -99 pid: 68711 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572516.734613:0:68711:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 6/2,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e08e9 expref: -99 pid: 68711 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572516.734620:0:68711:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 5/1,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e08e9 expref: -99 pid: 68711 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572516.734628:0:68711:0:(vvp_io.c:835:vvp_io_read_start()) f131a.replay-single: read [0, 1048576) 00000001:00010000:2.0:1788572516.734639:0:68711:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000a811:0x6:0x0] 00000002:00010000:2.0:1788572516.734644:0:68711:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000002:00010000:2.0:1788572516.734646:0:68711:0:(mdc_dev.c:1007:mdc_lock_init()) ### lock ffff94d4057bd9b0, mdc lock ffff94d40857b8f0, flags 400001002 00010000:00010000:2.0:1788572516.734653:0:68711:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 4/2,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 5 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e08e9 expref: -99 pid: 68711 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572516.734661:0:68711:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 4/2,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e08e9 expref: -99 pid: 68711 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1788572516.734668:0:68711:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788572516.734670:0:68711:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 7/2,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e08e9 expref: -99 pid: 68711 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572516.734682:0:68711:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 6/1,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e08e9 expref: -99 pid: 68711 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1788572516.734688:0:68711:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788572516.734693:0:68711:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000a811:0x6:0x0] updating i_size 8 i_blocks 8 00000020:00010000:2.0:1788572516.734695:0:68711:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff94d4120272e0 (ffff94d405cee368/0) at cl_lock_release():245 00000020:00010000:2.0:1788572516.734698:0:68711:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d4120272e0 (ffff94d405cee368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572516.734701:0:68711:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d4057bd9b0 (ffff94d405ceeea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572516.734704:0:68711:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d4120272e0 (ffff94d405cee368/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572516.734706:0:68711:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d4057bd9b0 (ffff94d405cee368/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572516.734712:0:68711:0:(vvp_io.c:893:vvp_io_read_start()) count:1048576 iocb pos:0 00000008:00010000:2.0:1788572516.734750:0:68711:0:(osc_cache.c:3010:osc_queue_sync_pages()) ### no dlm lock: extent: ffff94d405c47c00 00000008:00010000:2.0:1788572516.734789:0:56702:0:(osc_cache.c:2022:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff94d405c47c00 00010000:00010000:2.0:1788572516.734822:0:56702:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e08e9 expref: -99 pid: 68711 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00100000:2.0:1788572516.734857:0:56702:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff94d408c60700 - object 0x20000a811:6 offset 0<>4096 00000100:00100000:2.0:1788572516.734910:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488515584:192.168.202.125@tcp:3:cat.0 00000008:00010000:2.0:1788572516.755282:0:2405:0:(osc_cache.c:910:osc_extent_finish()) ### no dlm lock: extent: ffff94d405c47c00 00000020:00200000:2.0:1788572516.755301:0:2405:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1788572516.755381:0:2405:0:(osc_cache.c:496:__osc_extent_remove()) ### no dlm lock: extent: ffff94d405c47c00 00000100:00100000:2.0:1788572516.755430:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488515584:192.168.202.125@tcp:3:cat.0 00000020:00010000:2.0:1788572516.763013:0:68711:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff94d40392e218 (ffff94d405cee368/0) at cl_lock_release():245 00000020:00010000:2.0:1788572516.763026:0:68711:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d40392e218 (ffff94d405cee368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572516.763030:0:68711:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d4057bd7b0 (ffff94d405ceeea8/1) at cl_lock_cancel():142 00010000:00010000:2.0:1788572516.763042:0:68711:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e08e9 expref: -99 pid: 68711 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572516.763060:0:68711:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 3/0,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e08e9 expref: -99 pid: 68711 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:2.0:1788572516.763078:0:68711:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d40392e218 (ffff94d405cee368/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572516.763081:0:68711:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d4057bd7b0 (ffff94d405cee368/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572516.763094:0:68711:0:(vvp_io.c:533:vvp_io_advance()) advancing 8 bytes 00000080:00200000:2.0:1788572516.763137:0:68711:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x6:0x0] ignore/verify layout 0/0, layout version 2 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572516.763142:0:68711:0:(file.c:2238:ll_file_io_generic()) f131a.replay-single: 1 io complete with rc: 0, result: 8, restart: 0 00000080:00200000:2.0:1788572516.763147:0:68711:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 8 00000080:00200008:2.0:1788572516.763233:0:68711:0:(file.c:2483:do_file_read_iter()) START file f131a.replay-single:[0x20000a811:0x6:0x0], ppos: 8, count: 4194304 00000080:00200000:2.0:1788572516.763269:0:68711:0:(file.c:2090:ll_file_io_generic()) f131a.replay-single: read ppos: 8, bytes: 4194304 00000080:00200000:2.0:1788572516.763274:0:68711:0:(file.c:1848:ll_io_set_mirror()) f131a.replay-single: desiginated mirror: 0 00000020:00200000:2.0:1788572516.763281:0:68711:0:(cl_io.c:214:cl_io_rw_init()) header@00000000386dd75d[0x110, 2, [0x20000a811:0x6:0x0] hash] 00000020:00200000:2.0:1788572516.763283:0:68711:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [8, 4194312) 0 0 00000080:00200000:2.0:1788572516.763287:0:68711:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572516.763302:0:68711:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 8 chunk: [8, 1048576) 4194312, 1048568 00020000:00200000:2.0:1788572516.763307:0:68711:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572516.763333:0:68711:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 255] 00000002:00010000:2.0:1788572516.763346:0:68711:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000002:00010000:2.0:1788572516.763351:0:68711:0:(mdc_dev.c:1007:mdc_lock_init()) ### lock ffff94d4057bd7b0, mdc lock ffff94d40857a390, flags 0 00010000:00010000:2.0:1788572516.763369:0:68711:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 2/1,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 5 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e08e9 expref: -99 pid: 68711 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572516.763382:0:68711:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 2/1,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e08e9 expref: -99 pid: 68711 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572516.763394:0:68711:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 6/2,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e08e9 expref: -99 pid: 68711 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572516.763405:0:68711:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 6/2,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e08e9 expref: -99 pid: 68711 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572516.763413:0:68711:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 5/1,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e08e9 expref: -99 pid: 68711 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572516.763420:0:68711:0:(vvp_io.c:835:vvp_io_read_start()) f131a.replay-single: read [8, 1048576) 00000001:00010000:2.0:1788572516.763433:0:68711:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000a811:0x6:0x0] 00000002:00010000:2.0:1788572516.763438:0:68711:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000002:00010000:2.0:1788572516.763441:0:68711:0:(mdc_dev.c:1007:mdc_lock_init()) ### lock ffff94d4057bd9b0, mdc lock ffff94d40857b8f0, flags 400001002 00010000:00010000:2.0:1788572516.763448:0:68711:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 4/2,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 5 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e08e9 expref: -99 pid: 68711 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572516.763457:0:68711:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 4/2,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e08e9 expref: -99 pid: 68711 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1788572516.763465:0:68711:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788572516.763469:0:68711:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 7/2,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e08e9 expref: -99 pid: 68711 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572516.763476:0:68711:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 6/1,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e08e9 expref: -99 pid: 68711 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1788572516.763482:0:68711:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788572516.763488:0:68711:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000a811:0x6:0x0] updating i_size 8 i_blocks 8 00000020:00010000:2.0:1788572516.763491:0:68711:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff94d4120272e0 (ffff94d405cee368/0) at cl_lock_release():245 00000020:00010000:2.0:1788572516.763493:0:68711:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d4120272e0 (ffff94d405cee368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572516.763496:0:68711:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d4057bd9b0 (ffff94d405ceeea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572516.763499:0:68711:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d4120272e0 (ffff94d405cee368/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572516.763501:0:68711:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d4057bd9b0 (ffff94d405cee368/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572516.763507:0:68711:0:(vvp_io.c:893:vvp_io_read_start()) count:1048568 iocb pos:8 00000020:00010000:2.0:1788572516.763513:0:68711:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff94d40392e218 (ffff94d405cee368/0) at cl_lock_release():245 00000020:00010000:2.0:1788572516.763516:0:68711:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d40392e218 (ffff94d405cee368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572516.763518:0:68711:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d4057bd7b0 (ffff94d405ceeea8/1) at cl_lock_cancel():142 00010000:00010000:2.0:1788572516.763522:0:68711:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e08e9 expref: -99 pid: 68711 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572516.763541:0:68711:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 3/0,0 mode: PR/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e08e9 expref: -99 pid: 68711 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00010000:2.0:1788572516.763550:0:68711:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d40392e218 (ffff94d405cee368/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572516.763552:0:68711:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d4057bd7b0 (ffff94d405cee368/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572516.763557:0:68711:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000080:00200000:2.0:1788572516.763570:0:68711:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x6:0x0] ignore/verify layout 0/0, layout version 2 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572516.763573:0:68711:0:(file.c:2238:ll_file_io_generic()) f131a.replay-single: 1 io complete with rc: 0, result: 0, restart: 0 00000080:00200000:2.0:1788572516.763575:0:68711:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 0 00000080:00200008:2.0:1788572516.763685:0:68711:0:(file.c:479:ll_file_release()) START file f131a.replay-single:[0x20000a811:0x6:0x0](ffff94d4034963c8), flags 100000 00000002:00100000:2.0:1788572516.763747:0:68711:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff94d4049bed80 x1875450488514944/t107374182402(107374182402) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 736/824 e 0 to 0 dl 1788572532 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'cat.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572516.763759:0:68711:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572516.763765:0:68711:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c60700 x1875450488515840/t0(0) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/712 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572516.763790:0:68711:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job cat:9bf287c0-f582-4b83-84c1-15e766218620:68711:1875450488515840:192.168.202.125@tcp:35:cat.0 00000100:00100000:2.0:1788572516.764663:0:68711:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405411e40 going to sleep for 11 seconds 00000100:00100000:0.0:1788572516.772371:0:68711:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job cat:9bf287c0-f582-4b83-84c1-15e766218620:68711:1875450488515840:192.168.202.125@tcp:35:cat.0 00000002:00100000:0.0:1788572516.772433:0:68711:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff94d4049bed80 x1875450488514944/t107374182402(107374182402) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 736/824 e 0 to 0 dl 1788572532 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'cat.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1788572516.998438:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572517.580157:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488515968:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572517.580282:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488516224:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572517.580315:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488516480:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572517.580344:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:1.0:1788572517.580350:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488516096:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572517.580372:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488516352:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572517.588758:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488516096:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572517.588791:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488515968:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572517.588803:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488516224:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572517.613060:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488516480:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572517.613102:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488516352:192.168.202.125@tcp:400:kworker.0 00010000:00010000:2.0:1788572518.031418:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572519.045674:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572520.077444:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572521.095861:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572522.119866:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572522.693677:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488516608:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572522.693769:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488516864:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572522.693798:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488517120:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572522.693828:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:2.0:1788572522.693834:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488516736:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572522.693855:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488516992:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572522.701457:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488516736:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572522.701478:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488516992:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572522.701491:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488516608:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572522.701513:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488516864:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572522.701523:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488517120:192.168.202.125@tcp:400:kworker.0 00010000:00010000:0.0:1788572523.141520:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572524.165366:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788572524.360950:0:69277:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_131b skipping excluded test 131b 00010000:00010000:1.0:1788572525.194476:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788572526.177543:0:69429:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 132a: PFL new component instantiate replay ========================================================== 21:42:04 (1788572524) 00010000:00010000:0.0:1788572526.214156:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1788572526.479171:0:69444:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:3.0:1788572526.479242:0:69444:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572526.479264:0:69444:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1788572526.479281:0:69444:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:3.0:1788572526.479289:0:69444:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572526.479297:0:69444:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:3.0:1788572526.479301:0:69444:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0ac1 00000080:00010000:3.0:1788572526.479311:0:69444:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0ac1 from it ffffb68a41f97b58 00010000:00010000:3.0:1788572526.479315:0:69444:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1788572526.479324:0:69444:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:3.0:1788572526.479341:0:69444:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:3.0:1788572526.479356:0:69444:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f132a.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), flags=8 00000080:00200000:3.0:1788572526.479361:0:69444:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f132a.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), intent=getattr 00000002:00010000:3.0:1788572526.479375:0:69444:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f132a.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788572526.479420:0:69444:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0ae4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 69444 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572526.479428:0:69444:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0ae4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 69444 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572526.479439:0:69444:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0ae4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 69444 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572526.479465:0:69444:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69444:1875450488517248:192.168.202.125@tcp:101:lfs.0 00000100:00100000:3.0:1788572526.480047:0:69444:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405411480 going to sleep for 11 seconds 00000100:00100000:3.0:1788572526.497248:0:69444:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 107374182403 gen 4 00000100:00100000:3.0:1788572526.497267:0:69444:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69444:1875450488517248:192.168.202.125@tcp:101:lfs.0 00010000:00010000:3.0:1788572526.497287:0:69444:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0ae4 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 69444 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572526.497300:0:69444:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0ae4 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 69444 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572526.497307:0:69444:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0ae4 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 69444 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572526.497314:0:69444:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0ae4 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 69444 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572526.497323:0:69444:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0ae4 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 69444 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572526.497329:0:69444:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d400031e00 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:3.0:1788572526.497333:0:69444:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0ae4 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 69444 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572526.497341:0:69444:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0ae4 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 69444 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572526.497347:0:69444:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0ae4 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 69444 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572526.497354:0:69444:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572526.497360:0:69444:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0ae4 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 69444 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572526.497366:0:69444:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0ae4 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 69444 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572526.497388:0:69444:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff94d408c54a80 x1875450488517248/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 592/584 e 0 to 0 dl 1788572542 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788572526.497404:0:69444:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1788572526.497435:0:69444:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1788572526.497449:0:69444:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00010000:3.0:1788572526.497458:0:69444:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0ac1 from it ffffb68a41f97930 00010000:00010000:3.0:1788572526.497462:0:69444:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1788572526.497472:0:69444:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:2.0:1788572526.498034:0:69444:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572526.498055:0:69444:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572526.498064:0:69444:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:2.0:1788572526.498073:0:69444:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:2.0:1788572526.498080:0:69444:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572526.498086:0:69444:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572526.498090:0:69444:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0ac1 00000080:00010000:2.0:1788572526.498094:0:69444:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0ac1 from it ffffb68a41f97bf8 00010000:00010000:2.0:1788572526.498098:0:69444:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:2.0:1788572526.498106:0:69444:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:2.0:1788572526.498116:0:69444:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572526.498127:0:69444:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f132a.replay-single, flags=777 00000080:00200000:2.0:1788572526.498132:0:69444:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572526.498138:0:69444:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572526.498143:0:69444:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:2.0:1788572526.498152:0:69444:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:2.0:1788572526.498158:0:69444:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572526.498162:0:69444:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572526.498165:0:69444:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0ac1 00000080:00010000:2.0:1788572526.498168:0:69444:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0ac1 from it ffffb68a41f97b38 00010000:00010000:2.0:1788572526.498170:0:69444:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:2.0:1788572526.498178:0:69444:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:2.0:1788572526.498185:0:69444:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 3 00000080:00200000:2.0:1788572526.498189:0:69444:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f132a.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), file ffff94d416fe8f00, open_flags a141, mode 81b6 opened 0 00000080:00200000:2.0:1788572526.498198:0:69444:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f132a.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), intent=open|creat 00000002:00010000:2.0:1788572526.498470:0:69444:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f132a.replay-single,[0x20000a811:0x9:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0120502 00010000:00010000:2.0:1788572526.498483:0:69444:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:2.0:1788572526.498620:0:69444:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:2.0:1788572526.498628:0:69444:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572526.498631:0:69444:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0ac1 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4e08aa expref: -99 pid: 68225 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:2.0:1788572526.498667:0:69444:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537200/0x1e260726fbbc0aeb lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 69444 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572526.498674:0:69444:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537200/0x1e260726fbbc0aeb lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 69444 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572526.498685:0:69444:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572526.498697:0:69444:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c60a80 x1875450488517504/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 536/66408 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572526.498706:0:69444:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537200/0x1e260726fbbc0aeb lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 69444 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572526.498731:0:69444:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69444:1875450488517504:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572526.499662:0:69444:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d407e133c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572526.507451:0:69444:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69444:1875450488517504:192.168.202.125@tcp:101:lfs.0 00010000:00010000:1.0:1788572526.507472:0:69444:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537200/0x1e260726fbbc0aeb lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e091a expref: -99 pid: 69444 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572526.507508:0:69444:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537200/0x1e260726fbbc0aeb lrc: 4/0,1 mode: --/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e091a expref: -99 pid: 69444 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572526.507521:0:69444:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537200/0x1e260726fbbc0aeb lrc: 4/0,1 mode: --/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e091a expref: -99 pid: 69444 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572526.507532:0:69444:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537200/0x1e260726fbbc0aeb lrc: 4/0,1 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e091a expref: -99 pid: 69444 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572526.507543:0:69444:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537200/0x1e260726fbbc0aeb lrc: 4/0,1 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e091a expref: -99 pid: 69444 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572526.507557:0:69444:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537200/0x1e260726fbbc0aeb lrc: 5/1,1 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e091a expref: -99 pid: 69444 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572526.507565:0:69444:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537200/0x1e260726fbbc0aeb lrc: 5/1,1 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e091a expref: -99 pid: 69444 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572526.507573:0:69444:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537200/0x1e260726fbbc0aeb lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e091a expref: -99 pid: 69444 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572526.507585:0:69444:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x8000037, status=0 req@ffff94d408c60a80 x1875450488517504/t107374182404(107374182404) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 536/608 e 0 to 0 dl 1788572542 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572526.507598:0:69444:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d408c60a80 x1875450488517504/t107374182404(107374182404) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 536/608 e 0 to 0 dl 1788572542 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572526.507610:0:69444:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537200/0x1e260726fbbc0aeb lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e091a expref: -99 pid: 69444 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572526.507621:0:69444:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589977617/9 (0 0) 00000080:00200000:1.0:1788572526.507627:0:69444:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572526.507640:0:69444:0:(namei.c:1628:ll_create_it()) VFS Op:name=f132a.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), intent=open|creat 00000080:00200000:1.0:1788572526.507795:0:69444:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x20000a811:0x9:0x0](ffff94d410bc3248) 00000080:00200000:1.0:1788572526.507803:0:69444:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x20000a811:0x9:0x0], updating i_size 0 00000080:00200000:1.0:1788572526.507960:0:69444:0:(file.c:6826:ll_layout_conf()) [0x20000a811:0x9:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:1.0:1788572526.507971:0:69444:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x20000a811:0x9:0x0](ffff94d410bc3248) 00000080:00010000:1.0:1788572526.507975:0:69444:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a811:0x9:0x0](ffff94d410bc3248) for lock 0x1e260726fbbc0aeb 00000080:00010000:1.0:1788572526.507985:0:69444:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a811:0x9:0x0](ffff94d410bc3248) 00000080:00200008:1.0:1788572526.507999:0:69444:0:(file.c:1090:ll_file_open()) START file f132a.replay-single:[0x20000a811:0x9:0x0](ffff94d410bc3248), flags 120501 00000002:00100000:1.0:1788572526.508020:0:69444:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d408c60a80 x1875450488517504/t107374182404(107374182404) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 536/608 e 0 to 0 dl 1788572542 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788572526.508059:0:69444:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0aeb from it ffff94d408cf6ba0 00010000:00010000:1.0:1788572526.508063:0:69444:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537200/0x1e260726fbbc0aeb lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4e091a expref: -99 pid: 69444 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572526.508072:0:69444:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537200/0x1e260726fbbc0aeb lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e091a expref: -99 pid: 69444 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572526.508093:0:69444:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x20000a811:0x9:0x0](ffff94d410bc3248), inode mode 81a4 mask 40 00000080:00200000:1.0:1788572526.508145:0:69444:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x20000a811:0x9:0x0](ffff94d410bc3248), inode mode 81a4 mask 2 00000080:00200000:1.0:1788572526.508155:0:69444:0:(xattr.c:413:ll_xattr_set()) VFS Op:inode=[0x20000a811:0x9:0x0](ffff94d410bc3248), xattr lov 00000002:00010000:1.0:1788572526.508171:0:69444:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x20000a811:0x9:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 040000002 00010000:00010000:1.0:1788572526.508367:0:69444:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 69444 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572526.508376:0:69444:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 69444 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572526.508386:0:69444:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572526.508403:0:69444:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d4049bd180 x1875450488517760/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 736/131944 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572526.508413:0:69444:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 69444 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572526.508433:0:69444:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69444:1875450488517760:192.168.202.125@tcp:101:lfs.0 00000100:00100000:1.0:1788572526.508607:0:69444:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408c8b600 going to sleep for 11 seconds 00000100:00100000:2.0:1788572526.518126:0:2397:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.125@tcp (source: 12345-192.168.202.125@tcp) 00000100:00100000:3.0:1788572526.518262:0:42439:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875450511513728 00000100:00100000:3.0:1788572526.518272:0:42439:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875450511513728 00000100:00100000:3.0:1788572526.518327:0:42439:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.125@tcp, seq: 94 00000100:00100000:3.0:1788572526.518337:0:42439:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff94d408c60380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:9bf287c0-f582-4b83-84c1-15e766218620+5:66973:x1875450511513728:12345-192.168.202.125@tcp:104: 00010000:00010000:3.0:1788572526.518366:0:42439:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537200/0x1e260726fbbc0aeb lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xec901abfeb4e091a expref: -99 pid: 69444 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572526.518446:0:42439:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788572526.518454:0:42439:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff94d408c60380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:9bf287c0-f582-4b83-84c1-15e766218620+5:66973:x1875450511513728:12345-192.168.202.125@tcp:104: Request processed in 118us (338us total) trans 0 rc 0/0 00000100:00100000:3.0:1788572526.518463:0:42439:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.125@tcp, seq: 94 00010000:00010000:0.0:1788572526.518945:0:42448:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788572526.518953:0:42448:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537200/0x1e260726fbbc0aeb lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xec901abfeb4e091a expref: -99 pid: 69444 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572526.518967:0:42448:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412537200 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:0.0:1788572526.518972:0:42448:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537200/0x1e260726fbbc0aeb lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xec901abfeb4e091a expref: -99 pid: 69444 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572526.518982:0:42448:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537200/0x1e260726fbbc0aeb lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xec901abfeb4e091a expref: -99 pid: 69444 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572526.518991:0:42448:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537200/0x1e260726fbbc0aeb lrc: 3/0,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0xec901abfeb4e091a expref: -99 pid: 69444 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572526.519042:0:42448:0:(vvp_object.c:117:vvp_conf_set()) [0x20000a811:0x9:0x0]: losing layout lock 00010000:00010000:0.0:1788572526.519116:0:42448:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537200/0x1e260726fbbc0aeb lrc: 2/0,0 mode: --/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xec901abfeb4e091a expref: -99 pid: 69444 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572526.519125:0:42448:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572526.519160:0:42448:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537200/0x1e260726fbbc0aeb lrc: 1/0,0 mode: --/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xec901abfeb4e091a expref: -99 pid: 69444 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572526.519169:0:42448:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537200/0x1e260726fbbc0aeb lrc: 0/0,0 mode: --/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xec901abfeb4e091a expref: -99 pid: 69444 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572526.519190:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572526.519192:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572526.519194:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572526.519243:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1788572526.519259:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488517888:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572526.527747:0:69444:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69444:1875450488517760:192.168.202.125@tcp:101:lfs.0 00010000:00010000:1.0:1788572526.527771:0:69444:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af2 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 69444 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572526.527784:0:69444:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af2 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 69444 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572526.527796:0:69444:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af2 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 69444 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572526.527803:0:69444:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af2 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 69444 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572526.527811:0:69444:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af2 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 69444 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572526.527817:0:69444:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412583e00 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:1.0:1788572526.527822:0:69444:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af2 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 69444 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572526.527829:0:69444:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af2 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 69444 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572526.527836:0:69444:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af2 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 69444 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572526.527842:0:69444:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572526.527849:0:69444:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af2 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 69444 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572526.527856:0:69444:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af2 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 69444 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572526.527880:0:69444:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x800002b, status=0 req@ffff94d4049bd180 x1875450488517760/t107374182405(107374182405) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 736/848 e 0 to 0 dl 1788572542 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572526.527893:0:69444:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d4049bd180 x1875450488517760/t107374182405(107374182405) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 736/848 e 0 to 0 dl 1788572542 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572526.527910:0:69444:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572526.527999:0:69444:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788572526.528008:0:69444:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572526.528122:0:69444:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x20000a811:0x9:0x0], updating i_size 0 00000002:00100000:1.0:1788572526.528141:0:69444:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d4049bd180 x1875450488517760/t107374182405(107374182405) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572542 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572526.528177:0:69444:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff94d4049bd180 x1875450488517760/t107374182405(107374182405) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572542 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572526.528187:0:69444:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572526.528193:0:69444:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d4049bc380 x1875450488518144/t0(0) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/712 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572526.528261:0:69444:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69444:1875450488518144:192.168.202.125@tcp:35:lfs.0 00000100:00100000:1.0:1788572526.528324:0:69444:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408c8b840 going to sleep for 11 seconds 00000100:00100000:0.0:1788572526.528458:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488517888:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572526.536617:0:69444:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69444:1875450488518144:192.168.202.125@tcp:35:lfs.0 00000002:00100000:1.0:1788572526.536654:0:69444:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff94d4049bd180 x1875450488517760/t107374182405(107374182405) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572542 ref 2 fl Complete:RQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1788572526.536709:0:69444:0:(file.c:479:ll_file_release()) START file f132a.replay-single:[0x20000a811:0x9:0x0](ffff94d410bc3248), flags 100001 00000002:00100000:1.0:1788572526.536763:0:69444:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff94d408c60a80 x1875450488517504/t107374182404(107374182404) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 536/608 e 0 to 0 dl 1788572542 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572526.536774:0:69444:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572526.536779:0:69444:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d4049bf480 x1875450488518400/t0(0) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/712 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572526.536800:0:69444:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69444:1875450488518400:192.168.202.125@tcp:35:lfs.0 00000100:00100000:1.0:1788572526.536884:0:69444:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408c8bcc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572526.543633:0:69444:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69444:1875450488518400:192.168.202.125@tcp:35:lfs.0 00000002:00100000:1.0:1788572526.543670:0:69444:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff94d408c60a80 x1875450488517504/t107374182404(107374182404) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 536/608 e 0 to 0 dl 1788572542 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1788572527.238030:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:1.0:1788572527.483255:0:69453:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572527.483269:0:69453:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572527.483292:0:69453:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572527.483355:0:69453:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 69453 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572527.483365:0:69453:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 69453 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572527.483375:0:69453:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 69453 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572527.483402:0:69453:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69453:1875450488518528:192.168.202.125@tcp:101:lfs.0 00000100:00100000:1.0:1788572527.483494:0:69453:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408c8bcc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572527.489764:0:69453:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 107374182407 gen 4 00000100:00100000:1.0:1788572527.489848:0:69453:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69453:1875450488518528:192.168.202.125@tcp:101:lfs.0 00010000:00010000:1.0:1788572527.489871:0:69453:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572527.489883:0:69453:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572527.489900:0:69453:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572527.489911:0:69453:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572527.489924:0:69453:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572527.489932:0:69453:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572527.489939:0:69453:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572527.489951:0:69453:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d4049bce00 x1875450488518528/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572543 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572527.489966:0:69453:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572527.489977:0:69453:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788572527.489984:0:69453:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572527.490014:0:69453:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1788572527.490020:0:69453:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0af9 00000080:00010000:1.0:1788572527.490025:0:69453:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572527.490028:0:69453:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0af9 from it ffffb68a40d17cb8 00010000:00010000:1.0:1788572527.490032:0:69453:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572527.490040:0:69453:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788572527.499405:0:69453:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788572527.499421:0:69453:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572527.499448:0:69453:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572527.499466:0:69453:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572527.499825:0:69453:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572527.499837:0:69453:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572527.499845:0:69453:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572527.499849:0:69453:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0af9 00000080:00010000:2.0:1788572527.499857:0:69453:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0af9 from it ffffb68a40d17cb8 00010000:00010000:2.0:1788572527.499861:0:69453:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572527.499870:0:69453:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572527.500035:0:69453:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572527.500046:0:69453:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572527.500053:0:69453:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572527.500061:0:69453:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572527.500068:0:69453:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572527.500072:0:69453:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572527.500075:0:69453:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0af9 00000080:00010000:2.0:1788572527.500078:0:69453:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0af9 from it ffffb68a40d17bb0 00010000:00010000:2.0:1788572527.500082:0:69453:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572527.500088:0:69453:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572527.500369:0:69453:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 44 00000080:00200008:2.0:1788572527.500379:0:69453:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 100000 00000080:00200080:2.0:1788572527.500479:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:2.0:1788572527.500681:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541d400 uarg=0000000000000000 00800000:00000004:2.0:1788572527.500688:0:69453:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0000-mdc-ffff94d405cc3800: age 3670, max_age 3679 00000100:00100000:2.0:1788572527.500726:0:69453:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69453:1875450488518656:192.168.202.125@tcp:41:lfs.0 00000100:00100000:2.0:1788572527.501098:0:69453:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d407e130c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572527.505283:0:69453:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69453:1875450488518656:192.168.202.125@tcp:41:lfs.0 00800000:00000004:0.0:1788572527.505321:0:69453:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0000-mdc-ffff94d405cc3800: update ffff94d412909e10 cache blocks 317130/353529 objects 1023602/1024000 00000080:00200080:0.0:1788572527.505395:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.505414:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00800000:00000004:0.0:1788572527.505421:0:69453:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0001-mdc-ffff94d405cc3800: age 3670, max_age 3679 00000100:00100000:0.0:1788572527.505454:0:69453:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69453:1875450488518784:192.168.202.125@tcp:41:lfs.0 00000100:00100000:0.0:1788572527.505543:0:69453:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416e1f540 going to sleep for 11 seconds 00000100:00100000:0.0:1788572527.512276:0:69453:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69453:1875450488518784:192.168.202.125@tcp:41:lfs.0 00800000:00000004:0.0:1788572527.512317:0:69453:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0001-mdc-ffff94d405cc3800: update ffff94d407db9e10 cache blocks 317207/353529 objects 1023684/1024000 00000080:00200080:0.0:1788572527.512394:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512412:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512421:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512427:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512432:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512436:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512441:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512445:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512448:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512451:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512455:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512458:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512462:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512465:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512470:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512473:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512477:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512493:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512500:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512504:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512509:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512512:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512517:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512520:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512525:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512529:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512533:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512536:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512540:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512544:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512548:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512551:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512556:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512559:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512563:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512566:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512570:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512574:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512578:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512582:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512586:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512589:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512605:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512608:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512613:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512617:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512621:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512625:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512629:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512633:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512637:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512641:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512645:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512648:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512651:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512655:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512658:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512662:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512666:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512669:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512673:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512676:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512681:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512685:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512689:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512693:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512711:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512716:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512720:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512723:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512727:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512730:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512734:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512738:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512742:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512746:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512750:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512753:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512757:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512760:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512765:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512768:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512772:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512775:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512779:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512782:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512786:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512790:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512794:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512798:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512801:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512805:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512809:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512812:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512816:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512819:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512823:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512826:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512829:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512832:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512835:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512838:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512842:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512845:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512849:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512852:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512856:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512859:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512863:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512865:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512869:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512872:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512875:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512879:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512883:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512887:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512890:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512894:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512898:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512902:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512906:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512910:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512914:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512917:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512921:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00800000:00000080:0.0:1788572527.512924:0:69453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.512928:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.512935:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00020000:00000004:0.0:1788572527.512944:0:69453:0:(obd_class.h:1091:obd_statfs()) lustre-OST0000-osc-ffff94d405cc3800: age 3670, max_age 3679 00000100:00100000:0.0:1788572527.512983:0:69453:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69453:1875450488518912:192.168.202.125@tcp:13:lfs.0 00000100:00100000:0.0:1788572527.513062:0:69453:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416e1f540 going to sleep for 11 seconds 00000100:00100000:0.0:1788572527.518256:0:69453:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69453:1875450488518912:192.168.202.125@tcp:13:lfs.0 00020000:00000004:0.0:1788572527.518289:0:69453:0:(obd_class.h:1117:obd_statfs()) lustre-OST0000-osc-ffff94d405cc3800: update ffff94d407dbd190 cache blocks 901369/958279 objects 261733/262144 00000080:00200080:0.0:1788572527.518409:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.518427:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00020000:00000004:0.0:1788572527.518435:0:69453:0:(obd_class.h:1091:obd_statfs()) lustre-OST0001-osc-ffff94d405cc3800: age 3670, max_age 3679 00000100:00100000:0.0:1788572527.518470:0:69453:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69453:1875450488519040:192.168.202.125@tcp:13:lfs.0 00000100:00100000:0.0:1788572527.518551:0:69453:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416e1f540 going to sleep for 11 seconds 00000100:00100000:0.0:1788572527.522921:0:69453:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69453:1875450488519040:192.168.202.125@tcp:13:lfs.0 00020000:00000004:0.0:1788572527.522952:0:69453:0:(obd_class.h:1117:obd_statfs()) lustre-OST0001-osc-ffff94d405cc3800: update ffff94d407dbb7d0 cache blocks 901369/958279 objects 261733/262144 00000080:00200080:0.0:1788572527.526889:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.526923:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.526936:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.526940:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.526945:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.526949:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.526954:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.526958:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.526962:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.526965:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.526969:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.526972:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.526975:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.526978:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.526982:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.526985:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.526989:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.526991:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.526995:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.526998:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527001:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527005:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527009:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527013:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527016:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527020:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527024:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527027:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527031:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527035:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527038:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527042:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527045:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527048:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527052:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527055:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527058:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527061:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527064:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527067:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527071:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527073:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527077:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527079:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527083:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527085:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527089:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527091:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527095:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527110:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527115:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527119:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527122:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527126:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527129:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527133:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527136:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527139:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527143:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527146:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527150:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527153:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527157:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527160:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527164:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527167:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527170:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527173:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527177:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527180:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527183:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527186:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527190:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527193:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527197:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527200:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527230:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527234:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527239:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527242:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527246:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527249:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527253:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527256:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527259:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527261:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527264:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527267:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527270:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527273:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527276:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527279:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527282:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527285:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527289:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527291:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527295:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527297:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527308:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527312:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527317:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527320:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527324:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527327:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527331:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527335:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527338:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527341:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527345:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527348:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527352:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527355:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527358:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527361:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527364:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527367:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527370:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527373:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527377:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527380:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527384:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527388:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527392:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527397:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200080:0.0:1788572527.527400:0:69453:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffc5832ae50 00020000:00000080:0.0:1788572527.527403:0:69453:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00000080:00200008:0.0:1788572527.527416:0:69453:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 100000 00000080:00200000:0.0:1788572527.582439:0:69460:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:0.0:1788572527.582469:0:69460:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572527.582499:0:69460:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572527.582516:0:69460:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788572527.582523:0:69460:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572527.582531:0:69460:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572527.582535:0:69460:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0af9 00000080:00010000:0.0:1788572527.582543:0:69460:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0af9 from it ffffb68a41f87bd0 00010000:00010000:0.0:1788572527.582547:0:69460:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572527.582557:0:69460:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788572527.582574:0:69460:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572527.582588:0:69460:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg225-client.virtnet, dir=[0x200000007:0x1:0x0](ffff94d403679988), flags=1536 00000080:00200000:0.0:1788572527.582592:0:69460:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:0.0:1788572527.582598:0:69460:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572527.582604:0:69460:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788572527.582612:0:69460:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788572527.582618:0:69460:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572527.582622:0:69460:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572527.582624:0:69460:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0af9 00000080:00010000:0.0:1788572527.582627:0:69460:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0af9 from it ffffb68a41f87cd0 00010000:00010000:0.0:1788572527.582630:0:69460:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788572527.582637:0:69460:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1788572527.582644:0:69460:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 3 00000080:00200000:0.0:1788572527.582649:0:69460:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:0.0:1788572527.582654:0:69460:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572527.582659:0:69460:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1788572527.582665:0:69460:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1788572527.582671:0:69460:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572527.582674:0:69460:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572527.582676:0:69460:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0af9 00000080:00010000:0.0:1788572527.582679:0:69460:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0af9 from it ffffb68a41f87db0 00010000:00010000:0.0:1788572527.582682:0:69460:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1788572527.582689:0:69460:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:0.0:1788572527.582695:0:69460:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 3 00000080:00200000:0.0:1788572527.582700:0:69460:0:(namei.c:2129:ll_create_nd()) VFS Op:name=fsa-oleg225-client.virtnet, dir=[0x200000007:0x1:0x0](ffff94d403679988), flags=33188, excl=1 00000080:00200000:0.0:1788572527.582703:0:69460:0:(namei.c:2084:ll_mknod()) VFS Op:name=fsa-oleg225-client.virtnet, dir=[0x200000007:0x1:0x0](ffff94d403679988) mode 100644 dev 0 00010000:00010000:0.0:1788572527.582729:0:69460:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1788572527.582780:0:69460:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1788572527.582786:0:69460:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572527.582789:0:69460:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583e00/0x1e260726fbbc0af9 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4e0944 expref: -99 pid: 69453 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000020:00100000:0.0:1788572527.582811:0:69460:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788572527.582824:0:69460:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408d3dc00 x1875450488519296/t0(0) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 528/688 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572527.582849:0:69460:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job mcreate:9bf287c0-f582-4b83-84c1-15e766218620:69460:1875450488519296:192.168.202.125@tcp:36:mcreate.0 00000100:00100000:0.0:1788572527.582935:0:69460:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416e1f540 going to sleep for 11 seconds 00000100:00100000:0.0:1788572527.599017:0:69460:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job mcreate:9bf287c0-f582-4b83-84c1-15e766218620:69460:1875450488519296:192.168.202.125@tcp:36:mcreate.0 00000080:00200000:0.0:1788572527.599077:0:69460:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x20000a811:0xa:0x0](ffff94d410bc9988) 00000080:00200000:0.0:1788572527.599163:0:69460:0:(namei.c:2137:ll_create_nd()) VFS Op:name=fsa-oleg225-client.virtnet, unhashed 1 00000080:00200000:2.0:1788572527.617199:0:69463:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572527.617636:0:69463:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572527.617706:0:69463:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0b00 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 69463 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.617717:0:69463:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0b00 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 69463 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.617728:0:69463:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0b00 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 69463 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572527.617757:0:69463:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:69463:1875450488519424:192.168.202.125@tcp:101:rm.0 00000100:00100000:2.0:1788572527.617893:0:69463:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d407e13300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572527.621786:0:69463:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:69463:1875450488519424:192.168.202.125@tcp:101:rm.0 00010000:00010000:2.0:1788572527.621813:0:69463:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0b00 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0960 expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.621827:0:69463:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0b00 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0960 expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.621839:0:69463:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0b00 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0960 expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.621848:0:69463:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0b00 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0960 expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.621861:0:69463:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0b00 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0960 expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.621869:0:69463:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0b00 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0960 expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.621876:0:69463:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0b00 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0960 expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572527.621888:0:69463:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408c61f80 x1875450488519424/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572543 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572527.621902:0:69463:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0b00 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0960 expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.621915:0:69463:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788572527.621921:0:69463:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572527.621948:0:69463:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788572527.621953:0:69463:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0b00 00000080:00010000:2.0:1788572527.621958:0:69463:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572527.621961:0:69463:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b00 from it ffffb68a41e1fb58 00010000:00010000:2.0:1788572527.621965:0:69463:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0b00 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0960 expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.621974:0:69463:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0b00 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0960 expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572527.621999:0:69463:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572527.622018:0:69463:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg225-client.virtnet, dir=[0x200000007:0x1:0x0](ffff94d403679988), flags=8 00000080:00200000:2.0:1788572527.622022:0:69463:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=fsa-oleg225-client.virtnet, dir=[0x200000007:0x1:0x0](ffff94d403679988), intent=getattr 00000002:00010000:2.0:1788572527.622043:0:69463:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: fsa-oleg225-client.virtnet,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572527.622091:0:69463:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0b07 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 69463 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.622438:0:69463:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0b07 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 69463 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.622451:0:69463:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0b07 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 69463 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572527.622560:0:69463:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:69463:1875450488519552:192.168.202.125@tcp:101:rm.0 00000100:00100000:0.0:1788572527.622674:0:69463:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416e1f540 going to sleep for 11 seconds 00000100:00100000:0.0:1788572527.626790:0:69463:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:69463:1875450488519552:192.168.202.125@tcp:101:rm.0 00010000:00010000:0.0:1788572527.626817:0:69463:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0b07 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e097c expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572527.626851:0:69463:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0b07 lrc: 4/1,0 mode: --/PR res: [0x20000a811:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e097c expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572527.626860:0:69463:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0b07 lrc: 4/1,0 mode: --/PR res: [0x20000a811:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e097c expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572527.626870:0:69463:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0b07 lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e097c expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572527.626880:0:69463:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0b07 lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e097c expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572527.626892:0:69463:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0b07 lrc: 5/2,0 mode: PR/PR res: [0x20000a811:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e097c expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572527.626900:0:69463:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0b07 lrc: 5/2,0 mode: PR/PR res: [0x20000a811:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e097c expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572527.626907:0:69463:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0b07 lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e097c expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572527.626919:0:69463:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408c61f80 x1875450488519552/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 600/584 e 0 to 0 dl 1788572543 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572527.626932:0:69463:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0b07 lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e097c expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572527.626941:0:69463:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589977617/10 (0 0) 00000080:00200000:0.0:1788572527.626949:0:69463:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788572527.626970:0:69463:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x20000a811:0xa:0x0], updating i_size 0 00000080:00200000:0.0:1788572527.626973:0:69463:0:(namei.c:147:ll_iget()) got inode: [0x20000a811:0xa:0x0](ffff94d410bc9988): rc = 0 00000080:00200000:0.0:1788572527.627015:0:69463:0:(file.c:6826:ll_layout_conf()) [0x20000a811:0xa:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:0.0:1788572527.627039:0:69463:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a811:0xa:0x0](ffff94d410bc9988) for lock 0x1e260726fbbc0b07 00000080:00010000:0.0:1788572527.627050:0:69463:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a811:0xa:0x0](ffff94d410bc9988) 00000080:00010000:0.0:1788572527.627054:0:69463:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b07 from it ffffb68a41e1fb20 00010000:00010000:0.0:1788572527.627057:0:69463:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0b07 lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4e097c expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572527.627241:0:69463:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0b07 lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e097c expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1788572527.627281:0:69463:0:(file.c:6285:ll_getattr_dentry()) START file fsa-oleg225-client.virtnet:[0x20000a811:0xa:0x0](ffff94d410bc9988), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788572527.627288:0:69463:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x20000a811:0xa:0x0](ffff94d410bc9988),name=fsa-oleg225-client.virtnet 00000002:00010000:0.0:1788572527.627311:0:69463:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x20000a811:0xa:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572527.627320:0:69463:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0b07 lrc: 2/1,0 mode: PR/PR res: [0x20000a811:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e097c expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572527.627330:0:69463:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0b07 lrc: 2/1,0 mode: PR/PR res: [0x20000a811:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e097c expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572527.627338:0:69463:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572527.627444:0:69463:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a811:0xa:0x0](ffff94d410bc9988) 00000080:00010000:0.0:1788572527.627449:0:69463:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a811:0xa:0x0](ffff94d410bc9988) for lock 0x1e260726fbbc0b07 00000080:00010000:0.0:1788572527.627454:0:69463:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b07 from it ffffb68a41e1fcb8 00010000:00010000:0.0:1788572527.627458:0:69463:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0b07 lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4e097c expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572527.627465:0:69463:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0b07 lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e097c expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788572527.627532:0:69463:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:0.0:1788572527.627540:0:69463:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572527.627548:0:69463:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0b00 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0960 expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572527.627556:0:69463:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0b00 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0960 expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572527.627563:0:69463:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572527.627567:0:69463:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572527.627570:0:69463:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0b00 00000080:00010000:0.0:1788572527.627574:0:69463:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b00 from it ffffb68a41e1fc10 00010000:00010000:0.0:1788572527.627577:0:69463:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0b00 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0960 expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572527.627584:0:69463:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0b00 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0960 expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788572527.627593:0:69463:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572527.627602:0:69463:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=fsa-oleg225-client.virtnet, flags=0 00000080:00200000:0.0:1788572527.627607:0:69463:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:0.0:1788572527.627613:0:69463:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572527.627619:0:69463:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0b00 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0960 expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572527.627626:0:69463:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0b00 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0960 expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788572527.627633:0:69463:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572527.627637:0:69463:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572527.627639:0:69463:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0b00 00000080:00010000:0.0:1788572527.627642:0:69463:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b00 from it ffffb68a41e1fd68 00010000:00010000:0.0:1788572527.627645:0:69463:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0b00 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0960 expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572527.627652:0:69463:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0b00 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0960 expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788572527.627660:0:69463:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 3 00000080:00200000:0.0:1788572527.627664:0:69463:0:(namei.c:2418:ll_unlink()) VFS Op:name=fsa-oleg225-client.virtnet, dir=[0x200000007:0x1:0x0](ffff94d403679988) 00010000:00010000:0.0:1788572527.627701:0:69463:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0b00 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4e0960 expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572527.627724:0:69463:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0b07 lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xec901abfeb4e097c expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788572527.627739:0:69463:0:(vvp_object.c:117:vvp_conf_set()) [0x20000a811:0xa:0x0]: losing layout lock 00010000:00010000:0.0:1788572527.627776:0:69463:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0b07 lrc: 1/0,0 mode: --/PR res: [0x20000a811:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xec901abfeb4e097c expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572527.627784:0:69463:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0b00 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4e0960 expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572527.627790:0:69463:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788572527.627793:0:69463:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536a00/0x1e260726fbbc0b07 lrc: 0/0,0 mode: --/PR res: [0x20000a811:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xec901abfeb4e097c expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572527.627993:0:69463:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536e00/0x1e260726fbbc0b00 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4e0960 expref: -99 pid: 69463 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:0.0:1788572527.628012:0:69463:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788572527.628020:0:69463:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408d3f800 x1875450488519808/t0(0) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 512/736 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572527.628042:0:69463:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:69463:1875450488519808:192.168.202.125@tcp:36:rm.0 00000100:00100000:1.0:1788572527.630946:0:69463:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416e1f540 going to sleep for 11 seconds 00000100:00100000:1.0:1788572527.639536:0:69463:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job rm:9bf287c0-f582-4b83-84c1-15e766218620:69463:1875450488519808:192.168.202.125@tcp:36:rm.0 00000080:00200000:1.0:1788572527.639634:0:69463:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0xa:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788572527.639643:0:69463:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0xa:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572527.639667:0:69463:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x20000a811:0xa:0x0](ffff94d410bc9988) 00000100:00100000:2.0:1788572527.813618:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488520064:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572527.813709:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488520320:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572527.813740:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:2.0:1788572527.813746:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488519936:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572527.813770:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488520192:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572527.827946:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488519936:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572527.827970:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488520192:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572527.827981:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488520320:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572527.834702:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488520064:192.168.202.125@tcp:400:kworker.0 00010000:00010000:0.0:1788572528.261678:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572529.285397:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572530.312062:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572531.333502:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572532.358252:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572532.936545:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488520576:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572532.936638:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488520832:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572532.937412:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:0.0:1788572532.937424:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488520448:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572532.937456:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488520704:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572532.937493:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488520960:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572532.941616:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488520448:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572532.941640:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488520704:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572532.941726:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488520960:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572532.941741:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488520576:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572532.941753:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488520832:192.168.202.125@tcp:400:kworker.0 00010000:00010000:1.0:1788572533.381428:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572534.405364:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788572535.163034:0:69529:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 00000080:00200000:2.0:1788572535.188053:0:69530:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572535.188086:0:69530:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572535.189602:0:69530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536400/0x1e260726fbbc0b0e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 69530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572535.189620:0:69530:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536400/0x1e260726fbbc0b0e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 69530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572535.189631:0:69530:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536400/0x1e260726fbbc0b0e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 69530 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572535.189661:0:69530:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job dd:9bf287c0-f582-4b83-84c1-15e766218620:69530:1875450488521088:192.168.202.125@tcp:101:dd.0 00000100:00100000:2.0:1788572535.189756:0:69530:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d407e13300 going to sleep for 11 seconds 00000100:00100000:0.0:1788572535.192487:0:69530:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job dd:9bf287c0-f582-4b83-84c1-15e766218620:69530:1875450488521088:192.168.202.125@tcp:101:dd.0 00010000:00010000:0.0:1788572535.192513:0:69530:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536400/0x1e260726fbbc0b0e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09a6 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.192526:0:69530:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536400/0x1e260726fbbc0b0e lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09a6 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.192537:0:69530:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536400/0x1e260726fbbc0b0e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e09a6 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.192547:0:69530:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536400/0x1e260726fbbc0b0e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09a6 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.192559:0:69530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536400/0x1e260726fbbc0b0e lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e09a6 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.192567:0:69530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536400/0x1e260726fbbc0b0e lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e09a6 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.192575:0:69530:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536400/0x1e260726fbbc0b0e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e09a6 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572535.192585:0:69530:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408c60e00 x1875450488521088/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572551 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572535.192598:0:69530:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536400/0x1e260726fbbc0b0e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09a6 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.192609:0:69530:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788572535.192616:0:69530:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788572535.192643:0:69530:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1788572535.192648:0:69530:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0b0e 00000080:00010000:0.0:1788572535.192653:0:69530:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572535.192656:0:69530:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b0e from it ffffb68a4205fbf8 00010000:00010000:0.0:1788572535.192659:0:69530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536400/0x1e260726fbbc0b0e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e09a6 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.192667:0:69530:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536400/0x1e260726fbbc0b0e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09a6 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572535.192692:0:69530:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572535.192712:0:69530:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:0.0:1788572535.192726:0:69530:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572535.192734:0:69530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536400/0x1e260726fbbc0b0e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09a6 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572535.192749:0:69530:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536400/0x1e260726fbbc0b0e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09a6 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572535.192755:0:69530:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572535.192759:0:69530:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572535.192761:0:69530:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0b0e 00000080:00010000:0.0:1788572535.192764:0:69530:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b0e from it ffffb68a4205fb38 00010000:00010000:0.0:1788572535.192767:0:69530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536400/0x1e260726fbbc0b0e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e09a6 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572535.192775:0:69530:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536400/0x1e260726fbbc0b0e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09a6 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788572535.192782:0:69530:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 3 00000080:00200000:0.0:1788572535.192786:0:69530:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f132a.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), file ffff94d411038c00, open_flags 8042, mode 81b6 opened 0 00000080:00200000:0.0:1788572535.192793:0:69530:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f132a.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), intent=open|creat 00000002:00010000:0.0:1788572535.192809:0:69530:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f132a.replay-single,[0x20000a811:0xb:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788572535.192818:0:69530:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536400/0x1e260726fbbc0b0e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4e09a6 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572535.192856:0:69530:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536400/0x1e260726fbbc0b0e lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4e09a6 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572535.192863:0:69530:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572535.192866:0:69530:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536400/0x1e260726fbbc0b0e lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4e09a6 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572535.192900:0:69530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166200/0x1e260726fbbc0b15 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 69530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.192907:0:69530:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166200/0x1e260726fbbc0b15 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 69530 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572535.192917:0:69530:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572535.192923:0:69530:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408d3e680 x1875450488521344/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 776/66432 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572535.192933:0:69530:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166200/0x1e260726fbbc0b15 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 69530 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572535.192950:0:69530:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job dd:9bf287c0-f582-4b83-84c1-15e766218620:69530:1875450488521344:192.168.202.125@tcp:101:dd.0 00000100:00100000:0.0:1788572535.193058:0:69530:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416e1ff00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572535.198041:0:69530:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job dd:9bf287c0-f582-4b83-84c1-15e766218620:69530:1875450488521344:192.168.202.125@tcp:101:dd.0 00010000:00010000:0.0:1788572535.198080:0:69530:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166200/0x1e260726fbbc0b15 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09c2 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.198108:0:69530:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166200/0x1e260726fbbc0b15 lrc: 4/0,1 mode: --/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09c2 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.198117:0:69530:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166200/0x1e260726fbbc0b15 lrc: 4/0,1 mode: --/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09c2 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.198139:0:69530:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166200/0x1e260726fbbc0b15 lrc: 4/0,1 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e09c2 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.198155:0:69530:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166200/0x1e260726fbbc0b15 lrc: 4/0,1 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09c2 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.198167:0:69530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166200/0x1e260726fbbc0b15 lrc: 5/1,1 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e09c2 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.198175:0:69530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166200/0x1e260726fbbc0b15 lrc: 5/1,1 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e09c2 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.198193:0:69530:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166200/0x1e260726fbbc0b15 lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e09c2 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572535.198473:0:69530:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x2b, status=0 req@ffff94d408d3e680 x1875450488521344/t107374182410(107374182410) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 776/848 e 0 to 0 dl 1788572551 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788572535.198488:0:69530:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d408d3e680 x1875450488521344/t107374182410(107374182410) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 776/848 e 0 to 0 dl 1788572551 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572535.198501:0:69530:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166200/0x1e260726fbbc0b15 lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09c2 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788572535.198516:0:69530:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166200/0x1e260726fbbc0b15 lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09c2 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.198526:0:69530:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589977617/9 (0 0) 00000080:00200000:0.0:1788572535.198533:0:69530:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788572535.198635:0:69530:0:(namei.c:147:ll_iget()) got inode: [0x20000a811:0x9:0x0](ffff94d410bc3248): rc = 0 00000080:00200000:0.0:1788572535.198655:0:69530:0:(file.c:6826:ll_layout_conf()) [0x20000a811:0x9:0x0]: layout version change: 4294967294 -> 2 00000080:00010000:0.0:1788572535.198667:0:69530:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a811:0x9:0x0](ffff94d410bc3248) for lock 0x1e260726fbbc0b15 00000080:00010000:0.0:1788572535.198679:0:69530:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a811:0x9:0x0](ffff94d410bc3248) 00000080:00200008:0.0:1788572535.198694:0:69530:0:(file.c:1090:ll_file_open()) START file f132a.replay-single:[0x20000a811:0x9:0x0](ffff94d410bc3248), flags 100102 00000002:00100000:0.0:1788572535.198719:0:69530:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d408d3e680 x1875450488521344/t107374182410(107374182410) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 776/848 e 0 to 0 dl 1788572551 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788572535.198759:0:69530:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b15 from it ffff94d416ec4060 00010000:00010000:0.0:1788572535.198764:0:69530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166200/0x1e260726fbbc0b15 lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4e09c2 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.198772:0:69530:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166200/0x1e260726fbbc0b15 lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e09c2 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572535.198794:0:69530:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x20000a811:0x9:0x0](ffff94d410bc3248), inode mode 81a4 mask 46 00000002:00010000:0.0:1788572535.198846:0:69530:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x20000a811:0x9:0x0], intent: getxattr flags 0 00010000:00010000:0.0:1788572535.198888:0:69530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b1c lrc: 3/1,0 mode: --/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 69530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.198895:0:69530:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b1c lrc: 3/1,0 mode: --/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT flags: 0x0 pid: 69530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.198905:0:69530:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b1c lrc: 3/1,0 mode: --/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT flags: 0x0 pid: 69530 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572535.198927:0:69530:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job dd:9bf287c0-f582-4b83-84c1-15e766218620:69530:1875450488521472:192.168.202.125@tcp:101:dd.0 00000100:00100000:0.0:1788572535.199019:0:69530:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416e1ff00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572535.211564:0:69530:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job dd:9bf287c0-f582-4b83-84c1-15e766218620:69530:1875450488521472:192.168.202.125@tcp:101:dd.0 00010000:00010000:0.0:1788572535.211594:0:69530:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b1c lrc: 4/1,0 mode: --/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09d7 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.211615:0:69530:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b1c lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e09d7 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.211626:0:69530:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b1c lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09d7 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572535.211655:0:69530:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff94d408d3c380 x1875450488521472/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 568/1344 e 0 to 0 dl 1788572551 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572535.211668:0:69530:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b1c lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09d7 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.211682:0:69530:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589977617/9 (0 0) 00000080:00200000:0.0:1788572535.211689:0:69530:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x20000a811:0x9:0x0] suppgids 0 -1: rc 0 00000080:00010000:0.0:1788572535.211720:0:69530:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a811:0x9:0x0](ffff94d410bc3248) for lock 0x1e260726fbbc0b1c 00000080:00010000:0.0:1788572535.211728:0:69530:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b1c from it ffffb68a4205fc78 00010000:00010000:0.0:1788572535.211731:0:69530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b1c lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e09d7 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.211741:0:69530:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b1c lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09d7 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1788572535.211779:0:69530:0:(llite_lib.c:2436:ll_setattr_raw()) START file f132a.replay-single:[0x20000a811:0x9:0x0](ffff94d410bc3248) current size 0, valid attrs 2068, mode 0, uid 514, gid 0, new size 1048576, atime 1788572535.000000000, mtime 1788572535.000000000, ctime 1788572535.000000000 00010000:00010000:0.0:1788572535.211807:0:69530:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166200/0x1e260726fbbc0b15 lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xec901abfeb4e09c2 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572535.211861:0:69530:0:(vvp_object.c:117:vvp_conf_set()) [0x20000a811:0x9:0x0]: losing layout lock 00010000:00010000:0.0:1788572535.211919:0:69530:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166200/0x1e260726fbbc0b15 lrc: 1/0,0 mode: --/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xec901abfeb4e09c2 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572535.211927:0:69530:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572535.211930:0:69530:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166200/0x1e260726fbbc0b15 lrc: 0/0,0 mode: --/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xec901abfeb4e09c2 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788572535.211948:0:69530:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788572535.211955:0:69530:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408d3c380 x1875450488521728/t0(0) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572535.211979:0:69530:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job dd:9bf287c0-f582-4b83-84c1-15e766218620:69530:1875450488521728:192.168.202.125@tcp:36:dd.0 00000100:00100000:0.0:1788572535.212083:0:69530:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416e1ff00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572535.235610:0:69530:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job dd:9bf287c0-f582-4b83-84c1-15e766218620:69530:1875450488521728:192.168.202.125@tcp:36:dd.0 00000080:00200000:2.0:1788572535.239085:0:69530:0:(file.c:1848:ll_io_set_mirror()) f132a.replay-single: desiginated mirror: 0 00000080:00200000:2.0:1788572535.239109:0:69530:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00010000:00010000:2.0:1788572535.239132:0:69530:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 31 res 8589977617/9 (0 0) 00000080:00010000:2.0:1788572535.239143:0:69530:0:(file.c:7048:ll_layout_intent()) ### lustre: requeue layout lock for file [0x20000a811:0x9:0x0](ffff94d410bc3248) 00000002:00010000:2.0:1788572535.239154:0:69530:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x20000a811:0x9:0x0]) in obj [0x20000a811:0x9:0x0], intent: layout flags 0 00010000:00010000:2.0:1788572535.239255:0:69530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0b23 lrc: 3/1,0 mode: --/CR res: [0x20000a811:0x9:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 69530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572535.239266:0:69530:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0b23 lrc: 3/1,0 mode: --/CR res: [0x20000a811:0x9:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT flags: 0x0 pid: 69530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572535.239283:0:69530:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0b23 lrc: 3/1,0 mode: --/CR res: [0x20000a811:0x9:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT flags: 0x0 pid: 69530 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572535.239310:0:69530:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job dd:9bf287c0-f582-4b83-84c1-15e766218620:69530:1875450488521856:192.168.202.125@tcp:101:dd.0 00000100:00100000:2.0:1788572535.239401:0:69530:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d407e13000 going to sleep for 11 seconds 00000100:00100000:3.0:1788572535.249066:0:69530:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job dd:9bf287c0-f582-4b83-84c1-15e766218620:69530:1875450488521856:192.168.202.125@tcp:101:dd.0 00010000:00010000:3.0:1788572535.249112:0:69530:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0b23 lrc: 4/1,0 mode: CR/CR res: [0x20000a811:0x9:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e09e5 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572535.249129:0:69530:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0b23 lrc: 4/1,0 mode: CR/CR res: [0x20000a811:0x9:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09e5 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572535.249146:0:69530:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff94d408c61500 x1875450488521856/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 376/584 e 0 to 0 dl 1788572551 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572535.249157:0:69530:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0b23 lrc: 3/1,0 mode: CR/CR res: [0x20000a811:0x9:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09e5 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788572535.249178:0:69530:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0b23 lrc: 3/1,0 mode: CR/CR res: [0x20000a811:0x9:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09e5 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572535.249194:0:69530:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 16 res 8589977617/9 (0 0) 00000080:00200000:3.0:1788572535.249972:0:69530:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x20000a811:0x9:0x0] suppgids 0 0: rc 0 00000080:00010000:3.0:1788572535.250000:0:69530:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a811:0x9:0x0](ffff94d410bc3248) for lock 0x1e260726fbbc0b23 00000080:00010000:3.0:1788572535.250011:0:69530:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b23 from it ffffb68a4205fb08 00010000:00010000:3.0:1788572535.250015:0:69530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0b23 lrc: 3/1,0 mode: CR/CR res: [0x20000a811:0x9:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e09e5 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572535.250026:0:69530:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0b23 lrc: 2/0,0 mode: CR/CR res: [0x20000a811:0x9:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09e5 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572535.250064:0:69530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0b23 lrc: 2/1,0 mode: CR/CR res: [0x20000a811:0x9:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09e5 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572535.250076:0:69530:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0b23 lrc: 2/1,0 mode: CR/CR res: [0x20000a811:0x9:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09e5 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1788572535.250085:0:69530:0:(file.c:6931:ll_layout_lock_set()) ### file [0x20000a811:0x9:0x0](00000000f8dc364b) being reconfigured ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0b23 lrc: 3/1,0 mode: CR/CR res: [0x20000a811:0x9:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09e5 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572535.250152:0:69530:0:(file.c:6826:ll_layout_conf()) [0x20000a811:0x9:0x0]: layout version change: 4294967294 -> 2 00010000:00010000:3.0:1788572535.250172:0:69530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0b23 lrc: 3/1,0 mode: CR/CR res: [0x20000a811:0x9:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4e09e5 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572535.250181:0:69530:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0b23 lrc: 2/0,0 mode: CR/CR res: [0x20000a811:0x9:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e09e5 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572535.250199:0:69530:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x9:0x0] ignore/verify layout 0/0, layout version 2 need write layout 1, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572535.250401:0:69530:0:(vvp_io.c:371:vvp_io_fini()) [0x20000a811:0x9:0x0] write layout, type 3 [0x0, 0x100001) 00000080:00010000:3.0:1788572535.250409:0:69530:0:(file.c:7048:ll_layout_intent()) ### lustre: requeue layout lock for file [0x20000a811:0x9:0x0](ffff94d410bc3248) 00000002:00010000:3.0:1788572535.250420:0:69530:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x20000a811:0x9:0x0]) in obj [0x20000a811:0x9:0x0], intent: layout flags 02 00010000:00010000:3.0:1788572535.250435:0:69530:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0b23 lrc: 2/0,0 mode: CR/CR res: [0x20000a811:0x9:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xec901abfeb4e09e5 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572535.250470:0:69530:0:(vvp_object.c:117:vvp_conf_set()) [0x20000a811:0x9:0x0]: losing layout lock 00010000:00010000:3.0:1788572535.250509:0:69530:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0b23 lrc: 1/0,0 mode: --/CR res: [0x20000a811:0x9:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xec901abfeb4e09e5 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572535.250516:0:69530:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572535.250519:0:69530:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537a00/0x1e260726fbbc0b23 lrc: 0/0,0 mode: --/CR res: [0x20000a811:0x9:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xec901abfeb4e09e5 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572535.250544:0:69530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(EX) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0b2a lrc: 3/0,1 mode: --/EX res: [0x20000a811:0x9:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 69530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572535.250551:0:69530:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0b2a lrc: 3/0,1 mode: --/EX res: [0x20000a811:0x9:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT flags: 0x0 pid: 69530 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788572535.250563:0:69530:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788572535.250573:0:69530:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c54a80 x1875450488522112/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 376/584 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788572535.250583:0:69530:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0b2a lrc: 3/0,1 mode: --/EX res: [0x20000a811:0x9:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT flags: 0x0 pid: 69530 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572535.250603:0:69530:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job dd:9bf287c0-f582-4b83-84c1-15e766218620:69530:1875450488522112:192.168.202.125@tcp:101:dd.0 00000100:00100000:3.0:1788572535.250685:0:69530:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405411f00 going to sleep for 11 seconds 00000100:00100000:3.0:1788572535.274866:0:69530:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job dd:9bf287c0-f582-4b83-84c1-15e766218620:69530:1875450488522112:192.168.202.125@tcp:101:dd.0 00010000:00010000:3.0:1788572535.274895:0:69530:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode CR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0b2a lrc: 4/0,1 mode: --/EX res: [0x20000a811:0x9:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09f3 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572535.274907:0:69530:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0b2a lrc: 4/0,1 mode: --/CR res: [0x20000a811:0x9:0x0].0x0 bits 0xa/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09f3 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572535.274920:0:69530:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0b2a lrc: 4/0,1 mode: CR/CR res: [0x20000a811:0x9:0x0].0x0 bits 0xa/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e09f3 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572535.274930:0:69530:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0b2a lrc: 4/0,1 mode: CR/CR res: [0x20000a811:0x9:0x0].0x0 bits 0xa/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09f3 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572535.274944:0:69530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0b2a lrc: 5/1,1 mode: CR/CR res: [0x20000a811:0x9:0x0].0x0 bits 0xa/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e09f3 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572535.274953:0:69530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(EX) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0b2a lrc: 5/1,1 mode: CR/CR res: [0x20000a811:0x9:0x0].0x0 bits 0xa/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e09f3 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572535.274960:0:69530:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0b2a lrc: 4/1,0 mode: CR/CR res: [0x20000a811:0x9:0x0].0x0 bits 0xa/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e09f3 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572535.274971:0:69530:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff94d408c54a80 x1875450488522112/t107374182413(107374182413) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 376/608 e 0 to 0 dl 1788572551 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572535.274990:0:69530:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 264 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0b2a lrc: 3/1,0 mode: CR/CR res: [0x20000a811:0x9:0x0].0x0 bits 0xa/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09f3 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788572535.275005:0:69530:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0b2a lrc: 3/1,0 mode: CR/CR res: [0x20000a811:0x9:0x0].0x0 bits 0xa/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09f3 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572535.275018:0:69530:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 16 res 8589977617/9 (0 0) 00000080:00200000:3.0:1788572535.275025:0:69530:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x20000a811:0x9:0x0] suppgids 0 0: rc 0 00000080:00010000:3.0:1788572535.275049:0:69530:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a811:0x9:0x0](ffff94d410bc3248) for lock 0x1e260726fbbc0b2a 00000080:00010000:3.0:1788572535.275057:0:69530:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b2a from it ffffb68a4205fb70 00010000:00010000:3.0:1788572535.275060:0:69530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0b2a lrc: 3/1,0 mode: CR/CR res: [0x20000a811:0x9:0x0].0x0 bits 0xa/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e09f3 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572535.275068:0:69530:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0b2a lrc: 2/0,0 mode: CR/CR res: [0x20000a811:0x9:0x0].0x0 bits 0xa/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09f3 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572535.275086:0:69530:0:(file.c:1848:ll_io_set_mirror()) f132a.replay-single: desiginated mirror: 0 00000080:00200000:3.0:1788572535.275093:0:69530:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x9:0x0] ignore/verify layout 0/0, layout version 2 restore needed 0 00010000:00010000:3.0:1788572535.275108:0:69530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0b2a lrc: 2/1,0 mode: CR/CR res: [0x20000a811:0x9:0x0].0x0 bits 0xa/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09f3 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572535.275120:0:69530:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0b2a lrc: 2/1,0 mode: CR/CR res: [0x20000a811:0x9:0x0].0x0 bits 0xa/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09f3 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1788572535.275128:0:69530:0:(file.c:6931:ll_layout_lock_set()) ### file [0x20000a811:0x9:0x0](00000000f8dc364b) being reconfigured ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0b2a lrc: 3/1,0 mode: CR/CR res: [0x20000a811:0x9:0x0].0x0 bits 0xa/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09f3 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572535.275252:0:69530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0b2a lrc: 3/1,0 mode: CR/CR res: [0x20000a811:0x9:0x0].0x0 bits 0xa/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e09f3 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572535.275266:0:69530:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0b2a lrc: 2/0,0 mode: CR/CR res: [0x20000a811:0x9:0x0].0x0 bits 0xa/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09f3 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572535.275294:0:69530:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572535.275304:0:69530:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:2.0:1788572535.275730:0:69530:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x0 00000080:00200000:2.0:1788572535.275768:0:69530:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:2.0:1788572535.275799:0:69530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0b2a lrc: 2/1,0 mode: CR/CR res: [0x20000a811:0x9:0x0].0x0 bits 0xa/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09f3 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572535.275810:0:69530:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0b2a lrc: 2/1,0 mode: CR/CR res: [0x20000a811:0x9:0x0].0x0 bits 0xa/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09f3 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:2.0:1788572535.275817:0:69530:0:(file.c:6931:ll_layout_lock_set()) ### file [0x20000a811:0x9:0x0](00000000f8dc364b) being reconfigured ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0b2a lrc: 3/1,0 mode: CR/CR res: [0x20000a811:0x9:0x0].0x0 bits 0xa/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09f3 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572535.275866:0:69530:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572535.275871:0:69530:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:2.0:1788572535.275888:0:69530:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x0 00000080:00200000:2.0:1788572535.275905:0:69530:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572535.275985:0:69530:0:(file.c:6826:ll_layout_conf()) [0x20000a811:0x9:0x0]: layout version change: 4294967294 -> 3 00010000:00010000:2.0:1788572535.275995:0:69530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0b2a lrc: 3/1,0 mode: CR/CR res: [0x20000a811:0x9:0x0].0x0 bits 0xa/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4e09f3 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572535.276005:0:69530:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0b2a lrc: 2/0,0 mode: CR/CR res: [0x20000a811:0x9:0x0].0x0 bits 0xa/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e09f3 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00020000:00200000:2.0:1788572535.276022:0:69530:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:2.0:1788572535.276115:0:69530:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:2.0:1788572535.276136:0:69530:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1788572535.276141:0:69530:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff94d4164b2230, osc lock ffff94d40857a390, flags 0x0 00000008:00010000:2.0:1788572535.276145:0:69530:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1788572535.276147:0:69530:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff94d4164b2270, osc lock ffff94d40857b8f0, flags 0x0 00000008:00010000:2.0:1788572535.276150:0:69530:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1788572535.276151:0:69530:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff94d4164b22b0, osc lock ffff94d40857b430, flags 0x0 00010000:00010000:2.0:1788572535.276197:0:69530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572535.277897:0:69530:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->1048575] (req 0->1048575) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572535.277914:0:69530:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->1048575] (req 0->1048575) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572535.277950:0:69530:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job dd:9bf287c0-f582-4b83-84c1-15e766218620:69530:1875450488522240:192.168.202.125@tcp:101:dd.0 00000100:00100000:2.0:1788572535.279000:0:69530:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d407e130c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572535.289103:0:69530:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job dd:9bf287c0-f582-4b83-84c1-15e766218620:69530:1875450488522240:192.168.202.125@tcp:101:dd.0 00010000:00010000:0.0:1788572535.289134:0:69530:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 4/0,1 mode: --/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.289154:0:69530:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.289165:0:69530:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.289185:0:69530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1788572535.289198:0:69530:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.289251:0:69530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.289259:0:69530:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.289338:0:69530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.289348:0:69530:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [1048576->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.289358:0:69530:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [1048576->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572535.289380:0:69530:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job dd:9bf287c0-f582-4b83-84c1-15e766218620:69530:1875450488522368:192.168.202.125@tcp:101:dd.0 00000100:00100000:0.0:1788572535.289460:0:69530:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416e1ff00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572535.297016:0:69530:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job dd:9bf287c0-f582-4b83-84c1-15e766218620:69530:1875450488522368:192.168.202.125@tcp:101:dd.0 00010000:00010000:0.0:1788572535.297059:0:69530:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: --/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.297079:0:69530:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.297097:0:69530:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.297113:0:69530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 5/0,2 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1788572535.297126:0:69530:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,2 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.297136:0:69530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 5/0,2 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.297144:0:69530:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572535.297732:0:69530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x703:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572535.297746:0:69530:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x703:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572535.297757:0:69530:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 3/0,1 mode: --/PW res: [0x280000401:0x703:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572535.297781:0:69530:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job dd:9bf287c0-f582-4b83-84c1-15e766218620:69530:1875450488522496:192.168.202.125@tcp:101:dd.0 00000100:00100000:1.0:1788572535.297861:0:69530:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408c8b6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572535.303278:0:69530:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job dd:9bf287c0-f582-4b83-84c1-15e766218620:69530:1875450488522496:192.168.202.125@tcp:101:dd.0 00010000:00010000:1.0:1788572535.303316:0:69530:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x703:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0a08 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572535.303335:0:69530:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x703:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0a08 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572535.303351:0:69530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x703:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0a08 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1788572535.303365:0:69530:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x703:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4e0a08 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572535.303377:0:69530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x703:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a08 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572535.303386:0:69530:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x703:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a08 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572535.303529:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788572535.303548:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488522624:192.168.202.125@tcp:10:dd.0 00000100:00100000:1.0:1788572535.303795:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488522752:192.168.202.125@tcp:10:dd.0 00000100:00100000:1.0:1788572535.308564:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488522624:192.168.202.125@tcp:10:dd.0 00000100:00100000:1.0:1788572535.310427:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488522752:192.168.202.125@tcp:10:dd.0 00000080:00200000:3.0:1788572535.311518:0:69530:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000a811:0x9:0x0] updating i_size 1048576 i_blocks 0 00000020:00010000:3.0:1788572535.311528:0:69530:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff94d40392e218 (ffff94d405cee368/0) at cl_lock_release():245 00000020:00010000:3.0:1788572535.311533:0:69530:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d40392e218 (ffff94d405cee368/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788572535.311538:0:69530:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d4164b2230 (ffff94d405cee128/1) at cl_lock_cancel():142 00010000:00010000:3.0:1788572535.311549:0:69530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572535.311565:0:69530:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:3.0:1788572535.311588:0:69530:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d4164b2270 (ffff94d405ceeea8/1) at cl_lock_cancel():142 00010000:00010000:3.0:1788572535.311593:0:69530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572535.311601:0:69530:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:3.0:1788572535.311610:0:69530:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d4164b22b0 (ffff94d405ceec68/1) at cl_lock_cancel():142 00010000:00010000:3.0:1788572535.311615:0:69530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x703:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a08 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572535.311628:0:69530:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x703:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a08 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:3.0:1788572535.311635:0:69530:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d40392e218 (ffff94d405cee368/0) at cl_lock_fini():72 00000020:00010000:3.0:1788572535.311638:0:69530:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d4164b2230 (ffff94d405cee368/1) at cl_lock_fini():72 00000020:00010000:3.0:1788572535.311643:0:69530:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d4164b2270 (ffff94d405cee368/1) at cl_lock_fini():72 00000020:00010000:3.0:1788572535.311646:0:69530:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d4164b22b0 (ffff94d405cee368/1) at cl_lock_fini():72 00000080:00200000:3.0:1788572535.311707:0:69530:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x9:0x0] ignore/verify layout 0/0, layout version 3 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1788572535.311726:0:69530:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file f132a.replay-single:[0x20000a811:0x9:0x0](ffff94d410bc3248) current size 1048576, valid attrs 10002168, mode 0, uid 514, gid 0, new size 1048576, atime 1788572535.000000000, mtime 1788572535.000000000, ctime 1788572535.000000000, rc 0 00000080:00200080:3.0:1788572535.311773:0:69530:0:(file.c:4635:ll_file_ioctl()) VFS Op:inode=[0x20000a811:0x9:0x0](ffff94d410bc3248) cmd=80306d02 arg=7ffc9f242830 00020000:00000080:3.0:1788572535.311782:0:69530:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=80306d02 len=0 karg=0000000000000000 uarg=00007ffc9f242830 00000020:00000080:3.0:1788572535.311786:0:69530:0:(lov_obd.c:981:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: iocontrol from 'dd' cmd=80306d02 _IOR('m', 2, 48) unknown: rc = -25 00000080:00200008:3.0:1788572535.311792:0:69530:0:(file.c:5203:ll_file_seek()) START file f132a.replay-single:[0x20000a811:0x9:0x0], offset: 1048576, type: SEEK_CUR 00000080:00200008:3.0:1788572535.311798:0:69530:0:(file.c:5234:ll_file_seek()) COMPLETED file f132a.replay-single:[0x20000a811:0x9:0x0], offset: 1048576, type: SEEK_CUR, rc = 1048576 00000080:00200008:2.0:1788572535.340649:0:69530:0:(file.c:2640:do_file_write_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1048576, count: 1048576 00000080:00200000:2.0:1788572535.340705:0:69530:0:(file.c:2090:ll_file_io_generic()) f132a.replay-single: write ppos: 1048576, bytes: 1048576 00000080:00200000:2.0:1788572535.340711:0:69530:0:(file.c:1848:ll_io_set_mirror()) f132a.replay-single: desiginated mirror: 0 00000020:00200000:2.0:1788572535.340718:0:69530:0:(cl_io.c:214:cl_io_rw_init()) header@0000000099bd4c4d[0x110, 1, [0x20000a811:0x9:0x0] hash] 00000020:00200000:2.0:1788572535.340720:0:69530:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [1048576, 2097152) 0 0 00000080:00200000:2.0:1788572535.340725:0:69530:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572535.340738:0:69530:0:(file.c:2168:ll_file_io_generic()) Range lock [256, 511] 00020000:00200000:2.0:1788572535.340746:0:69530:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [1048576, 2097152) 2097152, 1048576 00020000:00200000:2.0:1788572535.340750:0:69530:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:2.0:1788572535.340778:0:69530:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [256, 511] 00000008:00010000:2.0:1788572535.340794:0:69530:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1788572535.340800:0:69530:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff94d4057bd730, osc lock ffff94d40857b7c0, flags 0x0 00010000:00010000:2.0:1788572535.340828:0:69530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572535.340850:0:69530:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (1048576 2097151) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572535.340864:0:69530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 6/0,2 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572535.340873:0:69530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 6/0,2 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572535.340881:0:69530:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 5/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572535.340968:0:69530:0:(vvp_io.c:1292:vvp_io_write_start()) f132a.replay-single: write [1048576, 2097152) append 0 append lockpos 0 00000080:00200000:2.0:1788572535.340982:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 256 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.341026:0:69530:0:(rw26.c:869:ll_write_end()) pos 1048576, len 4096, copied 4096 00000080:00200000:2.0:1788572535.341109:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000b91b66dc[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 256] 00000080:00200000:2.0:1788572535.341114:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000000234f38d 17ffffc0001001 3:0 ffff94d4124a12a0 256 lru 00000080:00200000:2.0:1788572535.341120:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000b9f84772 256: 1< 0 - - > 2< 1048576 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.341122:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000b91b66dc 00000080:00200000:2.0:1788572535.341124:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1788572535.341127:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 257 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.341147:0:69530:0:(rw26.c:869:ll_write_end()) pos 1052672, len 4096, copied 4096 00000080:00200000:2.0:1788572535.341152:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000080b21354[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 257] 00000080:00200000:2.0:1788572535.341154:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000007be823a6 17ffffc0001001 3:0 ffff94d4124a17e0 257 lru 00000080:00200000:2.0:1788572535.341160:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000005500452e 257: 1< 0 - - > 2< 1052672 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.341164:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000080b21354 00000080:00200000:2.0:1788572535.341165:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 2. 00000080:00200000:2.0:1788572535.341167:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 258 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.341184:0:69530:0:(rw26.c:869:ll_write_end()) pos 1056768, len 4096, copied 4096 00000080:00200000:2.0:1788572535.341189:0:69530:0:(rw26.c:927:ll_write_end()) page@000000006dd4ada0[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 258] 00000080:00200000:2.0:1788572535.341192:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000a28c108c 17ffffc0001001 3:0 ffff94d4124a1b60 258 lru 00000080:00200000:2.0:1788572535.341197:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000db234363 258: 1< 0 - - > 2< 1056768 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.341200:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000006dd4ada0 00000080:00200000:2.0:1788572535.341421:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 3. 00000080:00200000:2.0:1788572535.341424:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 259 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.341441:0:69530:0:(rw26.c:869:ll_write_end()) pos 1060864, len 4096, copied 4096 00000080:00200000:2.0:1788572535.341447:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000da03f601[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 259] 00000080:00200000:2.0:1788572535.341459:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000c0a007d0 17ffffc0001001 3:0 ffff94d4124a1a80 259 lru 00000080:00200000:2.0:1788572535.341464:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000071d6256b 259: 1< 0 - - > 2< 1060864 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.341466:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000da03f601 00000080:00200000:2.0:1788572535.341468:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 4. 00000080:00200000:2.0:1788572535.341470:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 260 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.341488:0:69530:0:(rw26.c:869:ll_write_end()) pos 1064960, len 4096, copied 4096 00000080:00200000:2.0:1788572535.341493:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000126501e3[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 260] 00000080:00200000:2.0:1788572535.341496:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000b538a098 17ffffc0001001 3:0 ffff94d4124a19a0 260 lru 00000080:00200000:2.0:1788572535.341501:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000b9ee9b77 260: 1< 0 - - > 2< 1064960 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.341502:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000126501e3 00000080:00200000:2.0:1788572535.341503:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 5. 00000080:00200000:2.0:1788572535.341505:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 261 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.341519:0:69530:0:(rw26.c:869:ll_write_end()) pos 1069056, len 4096, copied 4096 00000080:00200000:2.0:1788572535.341523:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000020c07518[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 261] 00000080:00200000:2.0:1788572535.341525:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000ce41dc64 17ffffc0001001 3:0 ffff94d4124a1d20 261 lru 00000080:00200000:2.0:1788572535.341530:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000020387aa 261: 1< 0 - - > 2< 1069056 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.341531:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000020c07518 00000080:00200000:2.0:1788572535.341532:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 6. 00000080:00200000:2.0:1788572535.341534:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 262 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.341556:0:69530:0:(rw26.c:869:ll_write_end()) pos 1073152, len 4096, copied 4096 00000080:00200000:2.0:1788572535.341560:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000059f089f8[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 262] 00000080:00200000:2.0:1788572535.341564:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000001119d444 17ffffc0001021 2:0 ffff94d4124a1700 262 lru 00000080:00200000:2.0:1788572535.341570:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000784f13f3 262: 1< 0 - - > 2< 1073152 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.341572:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000059f089f8 00000080:00200000:2.0:1788572535.341573:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 7. 00000080:00200000:2.0:1788572535.341574:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 263 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.341586:0:69530:0:(rw26.c:869:ll_write_end()) pos 1077248, len 4096, copied 4096 00000080:00200000:2.0:1788572535.341590:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000dea2922a[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 263] 00000080:00200000:2.0:1788572535.341593:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000046830ae2 17ffffc0001001 3:0 ffff94d4124a11c0 263 lru 00000080:00200000:2.0:1788572535.341598:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000c9c57376 263: 1< 0 - - > 2< 1077248 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.341600:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000dea2922a 00000080:00200000:2.0:1788572535.341601:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 8. 00000080:00200000:2.0:1788572535.341602:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 264 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.341615:0:69530:0:(rw26.c:869:ll_write_end()) pos 1081344, len 4096, copied 4096 00000080:00200000:2.0:1788572535.341620:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000985d9709[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 264] 00000080:00200000:2.0:1788572535.341623:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000065d25662 17ffffc0001001 3:0 ffff94d4124a1c40 264 lru 00000080:00200000:2.0:1788572535.341628:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000fbf6680c 264: 1< 0 - - > 2< 1081344 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.341630:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000985d9709 00000080:00200000:2.0:1788572535.341631:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 9. 00000080:00200000:2.0:1788572535.341633:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 265 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.341645:0:69530:0:(rw26.c:869:ll_write_end()) pos 1085440, len 4096, copied 4096 00000080:00200000:2.0:1788572535.341649:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000f558252d[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 265] 00000080:00200000:2.0:1788572535.341652:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000eab87750 17ffffc0001001 3:0 ffff94d4124a1380 265 lru 00000080:00200000:2.0:1788572535.341657:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000d7edffc2 265: 1< 0 - - > 2< 1085440 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.341659:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000f558252d 00000080:00200000:2.0:1788572535.341661:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 10. 00000080:00200000:2.0:1788572535.341662:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 266 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.341675:0:69530:0:(rw26.c:869:ll_write_end()) pos 1089536, len 4096, copied 4096 00000080:00200000:2.0:1788572535.341679:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000f10f10ca[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 266] 00000080:00200000:2.0:1788572535.341681:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000005e75ee6f 17ffffc0001001 3:0 ffff94d4124a1000 266 lru 00000080:00200000:2.0:1788572535.341687:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000009728d499 266: 1< 0 - - > 2< 1089536 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.341689:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000f10f10ca 00000080:00200000:2.0:1788572535.341690:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 11. 00000080:00200000:2.0:1788572535.341692:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 267 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.341707:0:69530:0:(rw26.c:869:ll_write_end()) pos 1093632, len 4096, copied 4096 00000080:00200000:2.0:1788572535.341711:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000381875a4[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 267] 00000080:00200000:2.0:1788572535.341714:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000274cdb8f 17ffffc0001001 3:0 ffff94d4124a1460 267 lru 00000080:00200000:2.0:1788572535.341718:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000009496d7d9 267: 1< 0 - - > 2< 1093632 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.341720:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000381875a4 00000080:00200000:2.0:1788572535.341721:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 12. 00000080:00200000:2.0:1788572535.341722:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 268 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.341735:0:69530:0:(rw26.c:869:ll_write_end()) pos 1097728, len 4096, copied 4096 00000080:00200000:2.0:1788572535.341739:0:69530:0:(rw26.c:927:ll_write_end()) page@000000008b09a5b0[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 268] 00000080:00200000:2.0:1788572535.341741:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000079fd9f67 17ffffc0001001 3:0 ffff94d4124a18c0 268 lru 00000080:00200000:2.0:1788572535.341746:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000c1b05de0 268: 1< 0 - - > 2< 1097728 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.341748:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000008b09a5b0 00000080:00200000:2.0:1788572535.341749:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 13. 00000080:00200000:2.0:1788572535.341751:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 269 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.341764:0:69530:0:(rw26.c:869:ll_write_end()) pos 1101824, len 4096, copied 4096 00000080:00200000:2.0:1788572535.341768:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000eab90787[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 269] 00000080:00200000:2.0:1788572535.341771:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000001e4bb1ff 17ffffc0001001 3:0 ffff94d4124a1ee0 269 lru 00000080:00200000:2.0:1788572535.341776:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000013546365 269: 1< 0 - - > 2< 1101824 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.341778:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000eab90787 00000080:00200000:2.0:1788572535.341779:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 14. 00000080:00200000:2.0:1788572535.341781:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 270 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.341794:0:69530:0:(rw26.c:869:ll_write_end()) pos 1105920, len 4096, copied 4096 00000080:00200000:2.0:1788572535.341798:0:69530:0:(rw26.c:927:ll_write_end()) page@000000008370cc82[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 270] 00000080:00200000:2.0:1788572535.341800:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000079a71900 17ffffc0001001 3:0 ffff94d4124a1540 270 lru 00000080:00200000:2.0:1788572535.341805:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000445c91a3 270: 1< 0 - - > 2< 1105920 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.341808:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000008370cc82 00000080:00200000:2.0:1788572535.341809:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 15. 00000080:00200000:2.0:1788572535.341811:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 271 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.341825:0:69530:0:(rw26.c:869:ll_write_end()) pos 1110016, len 4096, copied 4096 00000080:00200000:2.0:1788572535.341829:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000099d633b2[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 271] 00000080:00200000:2.0:1788572535.341832:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000025410af7 17ffffc0001001 3:0 ffff94d4124a1e00 271 lru 00000080:00200000:2.0:1788572535.341837:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000b53b5ea9 271: 1< 0 - - > 2< 1110016 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.341839:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000099d633b2 00000080:00200000:2.0:1788572535.341840:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 16. 00000080:00200000:2.0:1788572535.341841:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 272 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.341856:0:69530:0:(rw26.c:869:ll_write_end()) pos 1114112, len 4096, copied 4096 00000080:00200000:2.0:1788572535.341861:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000e38a32c3[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 272] 00000080:00200000:2.0:1788572535.341864:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000053270b19 17ffffc0001001 3:0 ffff94d4124a10e0 272 lru 00000080:00200000:2.0:1788572535.341869:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000443225a5 272: 1< 0 - - > 2< 1114112 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.341871:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000e38a32c3 00000080:00200000:2.0:1788572535.341872:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 17. 00000080:00200000:2.0:1788572535.341874:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 273 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.341890:0:69530:0:(rw26.c:869:ll_write_end()) pos 1118208, len 4096, copied 4096 00000080:00200000:2.0:1788572535.341964:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000c914b18c[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 273] 00000080:00200000:2.0:1788572535.341967:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000006e1aa370 17ffffc0001001 3:0 ffff94d405122620 273 lru 00000080:00200000:2.0:1788572535.341973:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000006985ab7e 273: 1< 0 - - > 2< 1118208 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.341975:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000c914b18c 00000080:00200000:2.0:1788572535.341976:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 18. 00000080:00200000:2.0:1788572535.341978:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 274 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.341998:0:69530:0:(rw26.c:869:ll_write_end()) pos 1122304, len 4096, copied 4096 00000080:00200000:2.0:1788572535.342002:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000ba0bc972[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 274] 00000080:00200000:2.0:1788572535.342005:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000be7b4d8d 17ffffc0001001 3:0 ffff94d4051229a0 274 lru 00000080:00200000:2.0:1788572535.342010:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000f361f71e 274: 1< 0 - - > 2< 1122304 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.342012:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000ba0bc972 00000080:00200000:2.0:1788572535.342013:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 19. 00000080:00200000:2.0:1788572535.342015:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 275 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.342030:0:69530:0:(rw26.c:869:ll_write_end()) pos 1126400, len 4096, copied 4096 00000080:00200000:2.0:1788572535.342105:0:69530:0:(rw26.c:927:ll_write_end()) page@000000004cfdc832[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 275] 00000080:00200000:2.0:1788572535.342109:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000124725d7 17ffffc0001001 3:0 ffff94d405122a80 275 lru 00000080:00200000:2.0:1788572535.342114:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000b4a2be22 275: 1< 0 - - > 2< 1126400 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.342116:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000004cfdc832 00000080:00200000:2.0:1788572535.342117:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 20. 00000080:00200000:2.0:1788572535.342119:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 276 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.342135:0:69530:0:(rw26.c:869:ll_write_end()) pos 1130496, len 4096, copied 4096 00000080:00200000:2.0:1788572535.342140:0:69530:0:(rw26.c:927:ll_write_end()) page@000000003bf868ba[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 276] 00000080:00200000:2.0:1788572535.342143:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000fc200818 17ffffc0001001 3:0 ffff94d405122700 276 lru 00000080:00200000:2.0:1788572535.342148:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000086e042b1 276: 1< 0 - - > 2< 1130496 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.342150:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000003bf868ba 00000080:00200000:2.0:1788572535.342150:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 21. 00000080:00200000:2.0:1788572535.342152:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 277 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.342169:0:69530:0:(rw26.c:869:ll_write_end()) pos 1134592, len 4096, copied 4096 00000080:00200000:2.0:1788572535.342174:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000fd1fc954[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 277] 00000080:00200000:2.0:1788572535.342176:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000089dd0366 17ffffc0001021 2:0 ffff94d4051221c0 277 lru 00000080:00200000:2.0:1788572535.342181:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000b28f8cfe 277: 1< 0 - - > 2< 1134592 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.342183:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000fd1fc954 00000080:00200000:2.0:1788572535.342184:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 22. 00000080:00200000:2.0:1788572535.342185:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 278 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.342199:0:69530:0:(rw26.c:869:ll_write_end()) pos 1138688, len 4096, copied 4096 00000080:00200000:2.0:1788572535.342381:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000ed308183[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 278] 00000080:00200000:2.0:1788572535.342385:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000046dd08e4 17ffffc0001001 3:0 ffff94d405122d20 278 lru 00000080:00200000:2.0:1788572535.342391:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000448247b9 278: 1< 0 - - > 2< 1138688 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.342393:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000ed308183 00000080:00200000:2.0:1788572535.342394:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 23. 00000080:00200000:2.0:1788572535.342396:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 279 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.342415:0:69530:0:(rw26.c:869:ll_write_end()) pos 1142784, len 4096, copied 4096 00000080:00200000:2.0:1788572535.342420:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000015d1a34e[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 279] 00000080:00200000:2.0:1788572535.342423:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000050de6d0a 17ffffc0001001 3:0 ffff94d405122380 279 lru 00000080:00200000:2.0:1788572535.342427:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000005040feec 279: 1< 0 - - > 2< 1142784 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.342431:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000015d1a34e 00000080:00200000:2.0:1788572535.342432:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 24. 00000080:00200000:2.0:1788572535.342434:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 280 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.342458:0:69530:0:(rw26.c:869:ll_write_end()) pos 1146880, len 4096, copied 4096 00000080:00200000:2.0:1788572535.342463:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000d774a19e[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 280] 00000080:00200000:2.0:1788572535.342465:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000522fda8b 17ffffc0001001 3:0 ffff94d405122000 280 lru 00000080:00200000:2.0:1788572535.342471:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000fb962835 280: 1< 0 - - > 2< 1146880 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.342473:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000d774a19e 00000080:00200000:2.0:1788572535.342474:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 25. 00000080:00200000:2.0:1788572535.342476:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 281 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.342490:0:69530:0:(rw26.c:869:ll_write_end()) pos 1150976, len 4096, copied 4096 00000080:00200000:2.0:1788572535.342494:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000098536f51[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 281] 00000080:00200000:2.0:1788572535.342496:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000006c8476c4 17ffffc0001001 3:0 ffff94d405122ee0 281 lru 00000080:00200000:2.0:1788572535.342501:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000bfc68eac 281: 1< 0 - - > 2< 1150976 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.342503:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000098536f51 00000080:00200000:2.0:1788572535.342504:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 26. 00000080:00200000:2.0:1788572535.342506:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 282 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.342516:0:69530:0:(rw26.c:869:ll_write_end()) pos 1155072, len 4096, copied 4096 00000080:00200000:2.0:1788572535.342520:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000964b93c0[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 282] 00000080:00200000:2.0:1788572535.342523:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000005729e676 17ffffc0001001 3:0 ffff94d405122e00 282 lru 00000080:00200000:2.0:1788572535.342528:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000053b3dbd6 282: 1< 0 - - > 2< 1155072 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.342531:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000964b93c0 00000080:00200000:2.0:1788572535.342531:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 27. 00000080:00200000:2.0:1788572535.342533:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 283 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.342545:0:69530:0:(rw26.c:869:ll_write_end()) pos 1159168, len 4096, copied 4096 00000080:00200000:2.0:1788572535.342549:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000079d5f529[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 283] 00000080:00200000:2.0:1788572535.342552:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000c3f6d964 17ffffc0001001 3:0 ffff94d4051222a0 283 lru 00000080:00200000:2.0:1788572535.342557:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000002550af79 283: 1< 0 - - > 2< 1159168 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.342559:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000079d5f529 00000080:00200000:2.0:1788572535.342560:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 28. 00000080:00200000:2.0:1788572535.342562:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 284 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.342572:0:69530:0:(rw26.c:869:ll_write_end()) pos 1163264, len 4096, copied 4096 00000080:00200000:2.0:1788572535.342575:0:69530:0:(rw26.c:927:ll_write_end()) page@000000006633e2ae[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 284] 00000080:00200000:2.0:1788572535.342577:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000cbda67b8 17ffffc0001001 3:0 ffff94d405122b60 284 lru 00000080:00200000:2.0:1788572535.342583:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000045a5adde 284: 1< 0 - - > 2< 1163264 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.342585:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000006633e2ae 00000080:00200000:2.0:1788572535.342586:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 29. 00000080:00200000:2.0:1788572535.342588:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 285 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.342602:0:69530:0:(rw26.c:869:ll_write_end()) pos 1167360, len 4096, copied 4096 00000080:00200000:2.0:1788572535.342606:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000787cb939[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 285] 00000080:00200000:2.0:1788572535.342608:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000b5e43803 17ffffc0001001 3:0 ffff94d405122c40 285 lru 00000080:00200000:2.0:1788572535.342613:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000cb4a007b 285: 1< 0 - - > 2< 1167360 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.342614:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000787cb939 00000080:00200000:2.0:1788572535.342615:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 30. 00000080:00200000:2.0:1788572535.342617:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 286 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.342629:0:69530:0:(rw26.c:869:ll_write_end()) pos 1171456, len 4096, copied 4096 00000080:00200000:2.0:1788572535.342633:0:69530:0:(rw26.c:927:ll_write_end()) page@000000003e3d64bb[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 286] 00000080:00200000:2.0:1788572535.342636:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000129bb086 17ffffc0001001 3:0 ffff94d405122460 286 lru 00000080:00200000:2.0:1788572535.342641:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000a78f9099 286: 1< 0 - - > 2< 1171456 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.342643:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000003e3d64bb 00000080:00200000:2.0:1788572535.342644:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 31. 00000080:00200000:2.0:1788572535.342646:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 287 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.342659:0:69530:0:(rw26.c:869:ll_write_end()) pos 1175552, len 4096, copied 4096 00000080:00200000:2.0:1788572535.342663:0:69530:0:(rw26.c:927:ll_write_end()) page@000000008b6e0ffc[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 287] 00000080:00200000:2.0:1788572535.342665:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000c7fe44f6 17ffffc0001001 3:0 ffff94d4051228c0 287 lru 00000080:00200000:2.0:1788572535.342670:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000ce15feb9 287: 1< 0 - - > 2< 1175552 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.342672:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000008b6e0ffc 00000080:00200000:2.0:1788572535.342673:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 32. 00000080:00200000:2.0:1788572535.342675:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 288 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.342689:0:69530:0:(rw26.c:869:ll_write_end()) pos 1179648, len 4096, copied 4096 00000080:00200000:2.0:1788572535.342693:0:69530:0:(rw26.c:927:ll_write_end()) page@000000000667a6fb[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 288] 00000080:00200000:2.0:1788572535.342696:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000e5c39200 17ffffc0001001 3:0 ffff94d405122540 288 lru 00000080:00200000:2.0:1788572535.342702:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000080bc488b 288: 1< 0 - - > 2< 1179648 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.342704:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000000667a6fb 00000080:00200000:2.0:1788572535.342705:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 33. 00000080:00200000:2.0:1788572535.342706:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 289 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.342717:0:69530:0:(rw26.c:869:ll_write_end()) pos 1183744, len 4096, copied 4096 00000080:00200000:2.0:1788572535.342721:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000fd4e9ea4[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 289] 00000080:00200000:2.0:1788572535.342724:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000003ecd31f6 17ffffc0001001 3:0 ffff94d4051220e0 289 lru 00000080:00200000:2.0:1788572535.342728:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000009efa7806 289: 1< 0 - - > 2< 1183744 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.342730:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000fd4e9ea4 00000080:00200000:2.0:1788572535.342731:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 34. 00000080:00200000:2.0:1788572535.342733:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 290 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.342743:0:69530:0:(rw26.c:869:ll_write_end()) pos 1187840, len 4096, copied 4096 00000080:00200000:2.0:1788572535.342746:0:69530:0:(rw26.c:927:ll_write_end()) page@000000008994bc44[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 290] 00000080:00200000:2.0:1788572535.342748:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000e4b11490 17ffffc0001001 3:0 ffff94d4051227e0 290 lru 00000080:00200000:2.0:1788572535.342752:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000b3b05edc 290: 1< 0 - - > 2< 1187840 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.342754:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000008994bc44 00000080:00200000:2.0:1788572535.342755:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 35. 00000080:00200000:2.0:1788572535.342756:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 291 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.342767:0:69530:0:(rw26.c:869:ll_write_end()) pos 1191936, len 4096, copied 4096 00000080:00200000:2.0:1788572535.342770:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000d9b56120[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 291] 00000080:00200000:2.0:1788572535.342773:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000ca3e79c8 17ffffc0001001 3:0 ffff94d412490000 291 lru 00000080:00200000:2.0:1788572535.342777:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000b74ef3b4 291: 1< 0 - - > 2< 1191936 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.342779:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000d9b56120 00000080:00200000:2.0:1788572535.342780:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 36. 00000080:00200000:2.0:1788572535.342782:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 292 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.342795:0:69530:0:(rw26.c:869:ll_write_end()) pos 1196032, len 4096, copied 4096 00000080:00200000:2.0:1788572535.342799:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000008e4f77b[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 292] 00000080:00200000:2.0:1788572535.342802:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000005e766873 17ffffc0001021 2:0 ffff94d412490ee0 292 lru 00000080:00200000:2.0:1788572535.342807:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000e5957d30 292: 1< 0 - - > 2< 1196032 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.342809:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000008e4f77b 00000080:00200000:2.0:1788572535.342811:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 37. 00000080:00200000:2.0:1788572535.342812:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 293 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.342823:0:69530:0:(rw26.c:869:ll_write_end()) pos 1200128, len 4096, copied 4096 00000080:00200000:2.0:1788572535.342827:0:69530:0:(rw26.c:927:ll_write_end()) page@000000000bd2bac7[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 293] 00000080:00200000:2.0:1788572535.342829:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000091aaeb93 17ffffc0001001 3:0 ffff94d412490e00 293 lru 00000080:00200000:2.0:1788572535.342835:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000002f8cc95f 293: 1< 0 - - > 2< 1200128 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.342837:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000000bd2bac7 00000080:00200000:2.0:1788572535.342838:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 38. 00000080:00200000:2.0:1788572535.342839:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 294 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.342852:0:69530:0:(rw26.c:869:ll_write_end()) pos 1204224, len 4096, copied 4096 00000080:00200000:2.0:1788572535.342855:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000330326ef[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 294] 00000080:00200000:2.0:1788572535.342858:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000006f65c6cd 17ffffc0001001 3:0 ffff94d4124902a0 294 lru 00000080:00200000:2.0:1788572535.342863:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000ca8c1442 294: 1< 0 - - > 2< 1204224 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.342865:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000330326ef 00000080:00200000:2.0:1788572535.342866:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 39. 00000080:00200000:2.0:1788572535.342868:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 295 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.342879:0:69530:0:(rw26.c:869:ll_write_end()) pos 1208320, len 4096, copied 4096 00000080:00200000:2.0:1788572535.342882:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000030d836f2[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 295] 00000080:00200000:2.0:1788572535.342885:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000edfae160 17ffffc0001001 3:0 ffff94d412490620 295 lru 00000080:00200000:2.0:1788572535.342890:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000056565739 295: 1< 0 - - > 2< 1208320 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.342892:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000030d836f2 00000080:00200000:2.0:1788572535.342893:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 40. 00000080:00200000:2.0:1788572535.343123:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 296 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.343141:0:69530:0:(rw26.c:869:ll_write_end()) pos 1212416, len 4096, copied 4096 00000080:00200000:2.0:1788572535.343146:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000015d1ec92[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 296] 00000080:00200000:2.0:1788572535.343148:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000230f9a7f 17ffffc0001001 3:0 ffff94d4124908c0 296 lru 00000080:00200000:2.0:1788572535.343153:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000f5ec3e46 296: 1< 0 - - > 2< 1212416 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.343155:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000015d1ec92 00000080:00200000:2.0:1788572535.343156:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 41. 00000080:00200000:2.0:1788572535.343157:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 297 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.343168:0:69530:0:(rw26.c:869:ll_write_end()) pos 1216512, len 4096, copied 4096 00000080:00200000:2.0:1788572535.343171:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000dc667810[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 297] 00000080:00200000:2.0:1788572535.343174:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000066045c48 17ffffc0001001 3:0 ffff94d412490540 297 lru 00000080:00200000:2.0:1788572535.343178:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000080323e65 297: 1< 0 - - > 2< 1216512 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.343180:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000dc667810 00000080:00200000:2.0:1788572535.343181:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 42. 00000080:00200000:2.0:1788572535.343182:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 298 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.343191:0:69530:0:(rw26.c:869:ll_write_end()) pos 1220608, len 4096, copied 4096 00000080:00200000:2.0:1788572535.343194:0:69530:0:(rw26.c:927:ll_write_end()) page@000000007ebae318[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 298] 00000080:00200000:2.0:1788572535.343197:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000099c0445d 17ffffc0001001 3:0 ffff94d4124900e0 298 lru 00000080:00200000:2.0:1788572535.343414:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000008468b786 298: 1< 0 - - > 2< 1220608 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.343417:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000007ebae318 00000080:00200000:2.0:1788572535.343418:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 43. 00000080:00200000:2.0:1788572535.343421:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 299 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.343436:0:69530:0:(rw26.c:869:ll_write_end()) pos 1224704, len 4096, copied 4096 00000080:00200000:2.0:1788572535.343441:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000395c9f13[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 299] 00000080:00200000:2.0:1788572535.343444:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000fde5f915 17ffffc0001001 3:0 ffff94d4124907e0 299 lru 00000080:00200000:2.0:1788572535.343458:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000c7e09396 299: 1< 0 - - > 2< 1224704 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.343461:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000395c9f13 00000080:00200000:2.0:1788572535.343462:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 44. 00000080:00200000:2.0:1788572535.343464:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 300 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.343478:0:69530:0:(rw26.c:869:ll_write_end()) pos 1228800, len 4096, copied 4096 00000080:00200000:2.0:1788572535.343482:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000c645e5b8[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 300] 00000080:00200000:2.0:1788572535.343485:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000001eb515da 17ffffc0001001 3:0 ffff94d4124901c0 300 lru 00000080:00200000:2.0:1788572535.343490:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000054c3d86e 300: 1< 0 - - > 2< 1228800 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.343492:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000c645e5b8 00000080:00200000:2.0:1788572535.343493:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 45. 00000080:00200000:2.0:1788572535.343495:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 301 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.343505:0:69530:0:(rw26.c:869:ll_write_end()) pos 1232896, len 4096, copied 4096 00000080:00200000:2.0:1788572535.343508:0:69530:0:(rw26.c:927:ll_write_end()) page@000000000a7f685b[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 301] 00000080:00200000:2.0:1788572535.343511:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000de402ffb 17ffffc0001001 3:0 ffff94d412490c40 301 lru 00000080:00200000:2.0:1788572535.343516:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000fd05acd1 301: 1< 0 - - > 2< 1232896 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.343520:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000000a7f685b 00000080:00200000:2.0:1788572535.343522:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 46. 00000080:00200000:2.0:1788572535.343523:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 302 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.343535:0:69530:0:(rw26.c:869:ll_write_end()) pos 1236992, len 4096, copied 4096 00000080:00200000:2.0:1788572535.343539:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000c1eed60d[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 302] 00000080:00200000:2.0:1788572535.343542:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000ffb7c192 17ffffc0001001 3:0 ffff94d412490b60 302 lru 00000080:00200000:2.0:1788572535.343546:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000087ce3421 302: 1< 0 - - > 2< 1236992 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.343548:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000c1eed60d 00000080:00200000:2.0:1788572535.343549:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 47. 00000080:00200000:2.0:1788572535.343550:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 303 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.343562:0:69530:0:(rw26.c:869:ll_write_end()) pos 1241088, len 4096, copied 4096 00000080:00200000:2.0:1788572535.343566:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000c3d9107e[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 303] 00000080:00200000:2.0:1788572535.343568:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000094379c81 17ffffc0001001 3:0 ffff94d412490460 303 lru 00000080:00200000:2.0:1788572535.343572:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000cfe84ed2 303: 1< 0 - - > 2< 1241088 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.343574:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000c3d9107e 00000080:00200000:2.0:1788572535.343575:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 48. 00000080:00200000:2.0:1788572535.343576:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 304 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.343587:0:69530:0:(rw26.c:869:ll_write_end()) pos 1245184, len 4096, copied 4096 00000080:00200000:2.0:1788572535.343591:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000c1c2f2b8[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 304] 00000080:00200000:2.0:1788572535.343593:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000a2fb707f 17ffffc0001001 3:0 ffff94d412490a80 304 lru 00000080:00200000:2.0:1788572535.343598:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000003de39da8 304: 1< 0 - - > 2< 1245184 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.343601:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000c1c2f2b8 00000080:00200000:2.0:1788572535.343602:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 49. 00000080:00200000:2.0:1788572535.343603:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 305 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.343614:0:69530:0:(rw26.c:869:ll_write_end()) pos 1249280, len 4096, copied 4096 00000080:00200000:2.0:1788572535.343618:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000020637d90[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 305] 00000080:00200000:2.0:1788572535.343620:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000041e58063 17ffffc0001001 3:0 ffff94d412490700 305 lru 00000080:00200000:2.0:1788572535.343625:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000bf6c9e86 305: 1< 0 - - > 2< 1249280 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.343627:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000020637d90 00000080:00200000:2.0:1788572535.343628:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 50. 00000080:00200000:2.0:1788572535.343630:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 306 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.343642:0:69530:0:(rw26.c:869:ll_write_end()) pos 1253376, len 4096, copied 4096 00000080:00200000:2.0:1788572535.343645:0:69530:0:(rw26.c:927:ll_write_end()) page@000000008825765f[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 306] 00000080:00200000:2.0:1788572535.343648:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000008f656a9d 17ffffc0001001 3:0 ffff94d4124909a0 306 lru 00000080:00200000:2.0:1788572535.343652:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000feece98e 306: 1< 0 - - > 2< 1253376 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.343654:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000008825765f 00000080:00200000:2.0:1788572535.343655:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 51. 00000080:00200000:2.0:1788572535.343656:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 307 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.343673:0:69530:0:(rw26.c:869:ll_write_end()) pos 1257472, len 4096, copied 4096 00000080:00200000:2.0:1788572535.343677:0:69530:0:(rw26.c:927:ll_write_end()) page@000000005f948209[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 307] 00000080:00200000:2.0:1788572535.343680:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000ed1fd884 17ffffc0001021 2:0 ffff94d412490d20 307 lru 00000080:00200000:2.0:1788572535.343686:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000021df1440 307: 1< 0 - - > 2< 1257472 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.343688:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000005f948209 00000080:00200000:2.0:1788572535.343689:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 52. 00000080:00200000:2.0:1788572535.343691:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 308 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.343704:0:69530:0:(rw26.c:869:ll_write_end()) pos 1261568, len 4096, copied 4096 00000080:00200000:2.0:1788572535.343708:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000839edca1[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 308] 00000080:00200000:2.0:1788572535.343711:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000009bcd313f 17ffffc0001001 3:0 ffff94d412490380 308 lru 00000080:00200000:2.0:1788572535.343716:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000fb3577a0 308: 1< 0 - - > 2< 1261568 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.343718:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000839edca1 00000080:00200000:2.0:1788572535.343719:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 53. 00000080:00200000:2.0:1788572535.343721:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 309 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.343734:0:69530:0:(rw26.c:869:ll_write_end()) pos 1265664, len 4096, copied 4096 00000080:00200000:2.0:1788572535.343739:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000004c491a0[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 309] 00000080:00200000:2.0:1788572535.343742:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000347679da 17ffffc0001001 3:0 ffff94d4085e5700 309 lru 00000080:00200000:2.0:1788572535.343747:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000084a9402e 309: 1< 0 - - > 2< 1265664 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.343749:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000004c491a0 00000080:00200000:2.0:1788572535.343750:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 54. 00000080:00200000:2.0:1788572535.343752:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 310 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.343764:0:69530:0:(rw26.c:869:ll_write_end()) pos 1269760, len 4096, copied 4096 00000080:00200000:2.0:1788572535.343768:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000b4c23ed3[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 310] 00000080:00200000:2.0:1788572535.343770:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000009423a5dd 17ffffc0001001 3:0 ffff94d4085e51c0 310 lru 00000080:00200000:2.0:1788572535.343775:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000037a41e4f 310: 1< 0 - - > 2< 1269760 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.343776:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000b4c23ed3 00000080:00200000:2.0:1788572535.343777:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 55. 00000080:00200000:2.0:1788572535.343779:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 311 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.343789:0:69530:0:(rw26.c:869:ll_write_end()) pos 1273856, len 4096, copied 4096 00000080:00200000:2.0:1788572535.343793:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000047cde046[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 311] 00000080:00200000:2.0:1788572535.343796:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000b0717e6b 17ffffc0001001 3:0 ffff94d4085e5c40 311 lru 00000080:00200000:2.0:1788572535.343801:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000028834b42 311: 1< 0 - - > 2< 1273856 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.343803:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000047cde046 00000080:00200000:2.0:1788572535.343804:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 56. 00000080:00200000:2.0:1788572535.343805:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 312 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.343819:0:69530:0:(rw26.c:869:ll_write_end()) pos 1277952, len 4096, copied 4096 00000080:00200000:2.0:1788572535.343823:0:69530:0:(rw26.c:927:ll_write_end()) page@000000004e51467f[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 312] 00000080:00200000:2.0:1788572535.343825:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000002847b21e 17ffffc0001001 3:0 ffff94d4085e5460 312 lru 00000080:00200000:2.0:1788572535.343830:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000f277e8f4 312: 1< 0 - - > 2< 1277952 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.343832:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000004e51467f 00000080:00200000:2.0:1788572535.343833:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 57. 00000080:00200000:2.0:1788572535.343834:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 313 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.343846:0:69530:0:(rw26.c:869:ll_write_end()) pos 1282048, len 4096, copied 4096 00000080:00200000:2.0:1788572535.343850:0:69530:0:(rw26.c:927:ll_write_end()) page@000000008734f15d[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 313] 00000080:00200000:2.0:1788572535.343852:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000009519062c 17ffffc0001001 3:0 ffff94d4085e58c0 313 lru 00000080:00200000:2.0:1788572535.343856:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000001420b627 313: 1< 0 - - > 2< 1282048 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.343858:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000008734f15d 00000080:00200000:2.0:1788572535.343859:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 58. 00000080:00200000:2.0:1788572535.343861:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 314 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.343871:0:69530:0:(rw26.c:869:ll_write_end()) pos 1286144, len 4096, copied 4096 00000080:00200000:2.0:1788572535.343875:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000effb3f58[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 314] 00000080:00200000:2.0:1788572535.343877:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000040c039c0 17ffffc0001001 3:0 ffff94d4085e59a0 314 lru 00000080:00200000:2.0:1788572535.343881:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000c158618e 314: 1< 0 - - > 2< 1286144 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.343883:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000effb3f58 00000080:00200000:2.0:1788572535.343885:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 59. 00000080:00200000:2.0:1788572535.343886:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 315 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.343984:0:69530:0:(rw26.c:869:ll_write_end()) pos 1290240, len 4096, copied 4096 00000080:00200000:2.0:1788572535.343989:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000075177dd5[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 315] 00000080:00200000:2.0:1788572535.343991:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000d9ff9b00 17ffffc0001001 3:0 ffff94d4085e5540 315 lru 00000080:00200000:2.0:1788572535.343996:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000e6dfb6d8 315: 1< 0 - - > 2< 1290240 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.343998:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000075177dd5 00000080:00200000:2.0:1788572535.343999:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 60. 00000080:00200000:2.0:1788572535.344001:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 316 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.344017:0:69530:0:(rw26.c:869:ll_write_end()) pos 1294336, len 4096, copied 4096 00000080:00200000:2.0:1788572535.344021:0:69530:0:(rw26.c:927:ll_write_end()) page@000000005e22d77d[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 316] 00000080:00200000:2.0:1788572535.344024:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000005bb7cbdb 17ffffc0001001 3:0 ffff94d4085e5d20 316 lru 00000080:00200000:2.0:1788572535.344029:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000001824744d 316: 1< 0 - - > 2< 1294336 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.344095:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000005e22d77d 00000080:00200000:2.0:1788572535.344097:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 61. 00000080:00200000:2.0:1788572535.344099:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 317 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.344114:0:69530:0:(rw26.c:869:ll_write_end()) pos 1298432, len 4096, copied 4096 00000080:00200000:2.0:1788572535.344119:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000ea9af2d3[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 317] 00000080:00200000:2.0:1788572535.344121:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000ae5593e8 17ffffc0001001 3:0 ffff94d4085e50e0 317 lru 00000080:00200000:2.0:1788572535.344126:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000385dbc86 317: 1< 0 - - > 2< 1298432 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.344128:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000ea9af2d3 00000080:00200000:2.0:1788572535.344129:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 62. 00000080:00200000:2.0:1788572535.344131:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 318 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.344141:0:69530:0:(rw26.c:869:ll_write_end()) pos 1302528, len 4096, copied 4096 00000080:00200000:2.0:1788572535.344145:0:69530:0:(rw26.c:927:ll_write_end()) page@000000001ba80ddc[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 318] 00000080:00200000:2.0:1788572535.344147:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000f5c15e8b 17ffffc0001001 3:0 ffff94d4085e5380 318 lru 00000080:00200000:2.0:1788572535.344152:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000002d581631 318: 1< 0 - - > 2< 1302528 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.344154:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000001ba80ddc 00000080:00200000:2.0:1788572535.344155:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 63. 00000080:00200000:2.0:1788572535.344156:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 319 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.344168:0:69530:0:(rw26.c:869:ll_write_end()) pos 1306624, len 4096, copied 4096 00000080:00200000:2.0:1788572535.344173:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000bcc20985[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 319] 00000080:00200000:2.0:1788572535.344175:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000003bfba36c 17ffffc0001001 3:0 ffff94d4085e57e0 319 lru 00000080:00200000:2.0:1788572535.344179:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000003cbd860a 319: 1< 0 - - > 2< 1306624 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.344181:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000bcc20985 00000080:00200000:2.0:1788572535.344181:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 64. 00000080:00200000:2.0:1788572535.344183:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 320 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.344299:0:69530:0:(rw26.c:869:ll_write_end()) pos 1310720, len 4096, copied 4096 00000080:00200000:2.0:1788572535.344305:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000c92bb499[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 320] 00000080:00200000:2.0:1788572535.344308:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000008b42016 17ffffc0001001 3:0 ffff94d4085e5000 320 lru 00000080:00200000:2.0:1788572535.344386:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000068386262 320: 1< 0 - - > 2< 1310720 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.344389:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000c92bb499 00000080:00200000:2.0:1788572535.344390:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 65. 00000080:00200000:2.0:1788572535.344393:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 321 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.344411:0:69530:0:(rw26.c:869:ll_write_end()) pos 1314816, len 4096, copied 4096 00000080:00200000:2.0:1788572535.344417:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000097137d32[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 321] 00000080:00200000:2.0:1788572535.344419:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000e54cd715 17ffffc0001001 3:0 ffff94d4085e5a80 321 lru 00000080:00200000:2.0:1788572535.344424:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000003c8422ff 321: 1< 0 - - > 2< 1314816 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.344427:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000097137d32 00000080:00200000:2.0:1788572535.344428:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 66. 00000080:00200000:2.0:1788572535.344430:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 322 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.344463:0:69530:0:(rw26.c:869:ll_write_end()) pos 1318912, len 4096, copied 4096 00000080:00200000:2.0:1788572535.344469:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000312572d7[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 322] 00000080:00200000:2.0:1788572535.344471:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000babf2596 17ffffc0001021 2:0 ffff94d4085e5ee0 322 lru 00000080:00200000:2.0:1788572535.344477:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000006093492c 322: 1< 0 - - > 2< 1318912 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.344479:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000312572d7 00000080:00200000:2.0:1788572535.344480:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 67. 00000080:00200000:2.0:1788572535.344482:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 323 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.344500:0:69530:0:(rw26.c:869:ll_write_end()) pos 1323008, len 4096, copied 4096 00000080:00200000:2.0:1788572535.344505:0:69530:0:(rw26.c:927:ll_write_end()) page@000000003ff5c45b[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 323] 00000080:00200000:2.0:1788572535.344508:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000a2587997 17ffffc0001001 3:0 ffff94d4085e5e00 323 lru 00000080:00200000:2.0:1788572535.344513:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000001e077193 323: 1< 0 - - > 2< 1323008 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.344516:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000003ff5c45b 00000080:00200000:2.0:1788572535.344517:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 68. 00000080:00200000:2.0:1788572535.344519:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 324 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.344533:0:69530:0:(rw26.c:869:ll_write_end()) pos 1327104, len 4096, copied 4096 00000080:00200000:2.0:1788572535.344536:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000fdae7a03[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 324] 00000080:00200000:2.0:1788572535.344539:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000006e0a4a6b 17ffffc0001001 3:0 ffff94d4085e52a0 324 lru 00000080:00200000:2.0:1788572535.344544:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000ef5d41cd 324: 1< 0 - - > 2< 1327104 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.344546:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000fdae7a03 00000080:00200000:2.0:1788572535.344547:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 69. 00000080:00200000:2.0:1788572535.344549:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 325 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.344562:0:69530:0:(rw26.c:869:ll_write_end()) pos 1331200, len 4096, copied 4096 00000080:00200000:2.0:1788572535.344566:0:69530:0:(rw26.c:927:ll_write_end()) page@000000004599ec79[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 325] 00000080:00200000:2.0:1788572535.344569:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000ed8e7504 17ffffc0001001 3:0 ffff94d4085e5b60 325 lru 00000080:00200000:2.0:1788572535.344574:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000002cce2860 325: 1< 0 - - > 2< 1331200 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.344576:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000004599ec79 00000080:00200000:2.0:1788572535.344577:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 70. 00000080:00200000:2.0:1788572535.344579:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 326 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.344593:0:69530:0:(rw26.c:869:ll_write_end()) pos 1335296, len 4096, copied 4096 00000080:00200000:2.0:1788572535.344598:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000074c29915[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 326] 00000080:00200000:2.0:1788572535.344601:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000004e4aa18e 17ffffc0001001 3:0 ffff94d4085e5620 326 lru 00000080:00200000:2.0:1788572535.344605:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000068211167 326: 1< 0 - - > 2< 1335296 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.344607:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000074c29915 00000080:00200000:2.0:1788572535.344609:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 71. 00000080:00200000:2.0:1788572535.344610:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 327 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.344622:0:69530:0:(rw26.c:869:ll_write_end()) pos 1339392, len 4096, copied 4096 00000080:00200000:2.0:1788572535.344626:0:69530:0:(rw26.c:927:ll_write_end()) page@000000002b6dd4f6[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 327] 00000080:00200000:2.0:1788572535.344629:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000adbed375 17ffffc0001001 3:0 ffff94d4125e0460 327 lru 00000080:00200000:2.0:1788572535.344634:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000004c5f2bb2 327: 1< 0 - - > 2< 1339392 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.344636:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000002b6dd4f6 00000080:00200000:2.0:1788572535.344637:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 72. 00000080:00200000:2.0:1788572535.344639:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 328 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.344650:0:69530:0:(rw26.c:869:ll_write_end()) pos 1343488, len 4096, copied 4096 00000080:00200000:2.0:1788572535.344655:0:69530:0:(rw26.c:927:ll_write_end()) page@000000002d3dd697[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 328] 00000080:00200000:2.0:1788572535.344658:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000007ebb6a06 17ffffc0001001 3:0 ffff94d4125e08c0 328 lru 00000080:00200000:2.0:1788572535.344662:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000d726eba2 328: 1< 0 - - > 2< 1343488 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.344664:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000002d3dd697 00000080:00200000:2.0:1788572535.344665:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 73. 00000080:00200000:2.0:1788572535.344667:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 329 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.344681:0:69530:0:(rw26.c:869:ll_write_end()) pos 1347584, len 4096, copied 4096 00000080:00200000:2.0:1788572535.344685:0:69530:0:(rw26.c:927:ll_write_end()) page@000000009bc4170d[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 329] 00000080:00200000:2.0:1788572535.344688:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000006e5cf531 17ffffc0001001 3:0 ffff94d4125e0540 329 lru 00000080:00200000:2.0:1788572535.344693:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000dcb80604 329: 1< 0 - - > 2< 1347584 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.344695:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000009bc4170d 00000080:00200000:2.0:1788572535.344697:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 74. 00000080:00200000:2.0:1788572535.344699:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 330 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.344716:0:69530:0:(rw26.c:869:ll_write_end()) pos 1351680, len 4096, copied 4096 00000080:00200000:2.0:1788572535.344720:0:69530:0:(rw26.c:927:ll_write_end()) page@000000005a073f88[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 330] 00000080:00200000:2.0:1788572535.344723:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000aa608474 17ffffc0001001 3:0 ffff94d4125e00e0 330 lru 00000080:00200000:2.0:1788572535.344728:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000008fc8d368 330: 1< 0 - - > 2< 1351680 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.344730:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000005a073f88 00000080:00200000:2.0:1788572535.344731:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 75. 00000080:00200000:2.0:1788572535.344732:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 331 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.344746:0:69530:0:(rw26.c:869:ll_write_end()) pos 1355776, len 4096, copied 4096 00000080:00200000:2.0:1788572535.344752:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000e4e27216[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 331] 00000080:00200000:2.0:1788572535.344754:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000cd7dbaa6 17ffffc0001001 3:0 ffff94d4125e0000 331 lru 00000080:00200000:2.0:1788572535.344758:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000231d1f4c 331: 1< 0 - - > 2< 1355776 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.344760:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000e4e27216 00000080:00200000:2.0:1788572535.344761:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 76. 00000080:00200000:2.0:1788572535.344763:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 332 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.344776:0:69530:0:(rw26.c:869:ll_write_end()) pos 1359872, len 4096, copied 4096 00000080:00200000:2.0:1788572535.344780:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000223b32a7[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 332] 00000080:00200000:2.0:1788572535.344782:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000002734c39d 17ffffc0001001 3:0 ffff94d4125e07e0 332 lru 00000080:00200000:2.0:1788572535.344787:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000002cf1d0a0 332: 1< 0 - - > 2< 1359872 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.344790:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000223b32a7 00000080:00200000:2.0:1788572535.344791:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 77. 00000080:00200000:2.0:1788572535.344793:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 333 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.344804:0:69530:0:(rw26.c:869:ll_write_end()) pos 1363968, len 4096, copied 4096 00000080:00200000:2.0:1788572535.344808:0:69530:0:(rw26.c:927:ll_write_end()) page@000000007e404419[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 333] 00000080:00200000:2.0:1788572535.344811:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000002ea74f48 17ffffc0001001 3:0 ffff94d4125e0ee0 333 lru 00000080:00200000:2.0:1788572535.344815:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000003085701a 333: 1< 0 - - > 2< 1363968 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.344817:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000007e404419 00000080:00200000:2.0:1788572535.344819:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 78. 00000080:00200000:2.0:1788572535.344820:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 334 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.344834:0:69530:0:(rw26.c:869:ll_write_end()) pos 1368064, len 4096, copied 4096 00000080:00200000:2.0:1788572535.344838:0:69530:0:(rw26.c:927:ll_write_end()) page@000000001cbbd297[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 334] 00000080:00200000:2.0:1788572535.344840:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000bb97559c 17ffffc0001001 3:0 ffff94d4125e0a80 334 lru 00000080:00200000:2.0:1788572535.344845:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000056b41b84 334: 1< 0 - - > 2< 1368064 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.344847:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000001cbbd297 00000080:00200000:2.0:1788572535.344848:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 79. 00000080:00200000:2.0:1788572535.344850:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 335 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.344862:0:69530:0:(rw26.c:869:ll_write_end()) pos 1372160, len 4096, copied 4096 00000080:00200000:2.0:1788572535.344867:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000035248ea4[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 335] 00000080:00200000:2.0:1788572535.344869:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000007e9a95d0 17ffffc0001001 3:0 ffff94d4125e0e00 335 lru 00000080:00200000:2.0:1788572535.344873:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000002a58a46e 335: 1< 0 - - > 2< 1372160 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.344875:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000035248ea4 00000080:00200000:2.0:1788572535.344876:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 80. 00000080:00200000:2.0:1788572535.344879:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 336 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.344892:0:69530:0:(rw26.c:869:ll_write_end()) pos 1376256, len 4096, copied 4096 00000080:00200000:2.0:1788572535.344897:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000a81090a3[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 336] 00000080:00200000:2.0:1788572535.344900:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000843ed9a6 17ffffc0001001 3:0 ffff94d4125e02a0 336 lru 00000080:00200000:2.0:1788572535.344905:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000bb281cd8 336: 1< 0 - - > 2< 1376256 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.344908:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000a81090a3 00000080:00200000:2.0:1788572535.344909:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 81. 00000080:00200000:2.0:1788572535.344911:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 337 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.344930:0:69530:0:(rw26.c:869:ll_write_end()) pos 1380352, len 4096, copied 4096 00000080:00200000:2.0:1788572535.344935:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000e09bd3fc[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 337] 00000080:00200000:2.0:1788572535.344938:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000eb9147ea 17ffffc0001021 2:0 ffff94d4125e0b60 337 lru 00000080:00200000:2.0:1788572535.344945:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000de96d810 337: 1< 0 - - > 2< 1380352 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.344947:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000e09bd3fc 00000080:00200000:2.0:1788572535.344948:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 82. 00000080:00200000:2.0:1788572535.344950:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 338 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.344965:0:69530:0:(rw26.c:869:ll_write_end()) pos 1384448, len 4096, copied 4096 00000080:00200000:2.0:1788572535.344970:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000a743678d[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 338] 00000080:00200000:2.0:1788572535.344973:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000fe2a67a1 17ffffc0001001 3:0 ffff94d4125e0620 338 lru 00000080:00200000:2.0:1788572535.344978:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000057f89ebe 338: 1< 0 - - > 2< 1384448 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.344981:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000a743678d 00000080:00200000:2.0:1788572535.344982:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 83. 00000080:00200000:2.0:1788572535.344984:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 339 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.344998:0:69530:0:(rw26.c:869:ll_write_end()) pos 1388544, len 4096, copied 4096 00000080:00200000:2.0:1788572535.345002:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000a12bca49[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 339] 00000080:00200000:2.0:1788572535.345006:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000d0322e62 17ffffc0001001 3:0 ffff94d4125e09a0 339 lru 00000080:00200000:2.0:1788572535.345011:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000e4d90b9b 339: 1< 0 - - > 2< 1388544 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.345014:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000a12bca49 00000080:00200000:2.0:1788572535.345015:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 84. 00000080:00200000:2.0:1788572535.345017:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 340 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.345030:0:69530:0:(rw26.c:869:ll_write_end()) pos 1392640, len 4096, copied 4096 00000080:00200000:2.0:1788572535.345112:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000cde82245[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 340] 00000080:00200000:2.0:1788572535.345114:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000faba4f8a 17ffffc0001001 3:0 ffff94d4125e0700 340 lru 00000080:00200000:2.0:1788572535.345119:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000042d759ab 340: 1< 0 - - > 2< 1392640 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.345121:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000cde82245 00000080:00200000:2.0:1788572535.345122:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 85. 00000080:00200000:2.0:1788572535.345124:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 341 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.345140:0:69530:0:(rw26.c:869:ll_write_end()) pos 1396736, len 4096, copied 4096 00000080:00200000:2.0:1788572535.345145:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000096ae023[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 341] 00000080:00200000:2.0:1788572535.345148:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000002d23a37e 17ffffc0001001 3:0 ffff94d4125e0d20 341 lru 00000080:00200000:2.0:1788572535.345152:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000007feacb3 341: 1< 0 - - > 2< 1396736 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.345154:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000096ae023 00000080:00200000:2.0:1788572535.345155:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 86. 00000080:00200000:2.0:1788572535.345158:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 342 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.345171:0:69530:0:(rw26.c:869:ll_write_end()) pos 1400832, len 4096, copied 4096 00000080:00200000:2.0:1788572535.345176:0:69530:0:(rw26.c:927:ll_write_end()) page@000000007e60c998[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 342] 00000080:00200000:2.0:1788572535.345178:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000c92b7848 17ffffc0001001 3:0 ffff94d4125e01c0 342 lru 00000080:00200000:2.0:1788572535.345183:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000005fa93b32 342: 1< 0 - - > 2< 1400832 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.345185:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000007e60c998 00000080:00200000:2.0:1788572535.345186:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 87. 00000080:00200000:2.0:1788572535.345188:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 343 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.345442:0:69530:0:(rw26.c:869:ll_write_end()) pos 1404928, len 4096, copied 4096 00000080:00200000:2.0:1788572535.345459:0:69530:0:(rw26.c:927:ll_write_end()) page@000000007b505ee7[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 343] 00000080:00200000:2.0:1788572535.345462:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000001db10c16 17ffffc0001001 3:0 ffff94d4125e0c40 343 lru 00000080:00200000:2.0:1788572535.345467:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000002ff6a002 343: 1< 0 - - > 2< 1404928 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.345469:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000007b505ee7 00000080:00200000:2.0:1788572535.345470:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 88. 00000080:00200000:2.0:1788572535.345472:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 344 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.345493:0:69530:0:(rw26.c:869:ll_write_end()) pos 1409024, len 4096, copied 4096 00000080:00200000:2.0:1788572535.345498:0:69530:0:(rw26.c:927:ll_write_end()) page@000000004a89b9ab[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 344] 00000080:00200000:2.0:1788572535.345500:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000fe4a7f99 17ffffc0001001 3:0 ffff94d4125e0380 344 lru 00000080:00200000:2.0:1788572535.345505:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000e648ca5e 344: 1< 0 - - > 2< 1409024 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.345507:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000004a89b9ab 00000080:00200000:2.0:1788572535.345508:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 89. 00000080:00200000:2.0:1788572535.345511:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 345 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.345525:0:69530:0:(rw26.c:869:ll_write_end()) pos 1413120, len 4096, copied 4096 00000080:00200000:2.0:1788572535.345788:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000020fa31e2[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 345] 00000080:00200000:2.0:1788572535.345791:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000bbdf682a 17ffffc0001001 3:0 ffff94d408ddb7e0 345 lru 00000080:00200000:2.0:1788572535.345796:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000098d8a20e 345: 1< 0 - - > 2< 1413120 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.345800:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000020fa31e2 00000080:00200000:2.0:1788572535.345801:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 90. 00000080:00200000:2.0:1788572535.345804:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 346 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.345821:0:69530:0:(rw26.c:869:ll_write_end()) pos 1417216, len 4096, copied 4096 00000080:00200000:2.0:1788572535.345825:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000098790e0a[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 346] 00000080:00200000:2.0:1788572535.345827:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000087d37265 17ffffc0001001 3:0 ffff94d408ddba80 346 lru 00000080:00200000:2.0:1788572535.345832:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000a1e5a7b6 346: 1< 0 - - > 2< 1417216 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.345834:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000098790e0a 00000080:00200000:2.0:1788572535.345835:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 91. 00000080:00200000:2.0:1788572535.345837:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 347 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.345850:0:69530:0:(rw26.c:869:ll_write_end()) pos 1421312, len 4096, copied 4096 00000080:00200000:2.0:1788572535.345854:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000f022fe3d[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 347] 00000080:00200000:2.0:1788572535.345857:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000bffabf96 17ffffc0001001 3:0 ffff94d408ddbb60 347 lru 00000080:00200000:2.0:1788572535.345862:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000e24c9e0b 347: 1< 0 - - > 2< 1421312 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.345864:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000f022fe3d 00000080:00200000:2.0:1788572535.345865:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 92. 00000080:00200000:2.0:1788572535.345867:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 348 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.345881:0:69530:0:(rw26.c:869:ll_write_end()) pos 1425408, len 4096, copied 4096 00000080:00200000:2.0:1788572535.345885:0:69530:0:(rw26.c:927:ll_write_end()) page@000000002abf9b14[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 348] 00000080:00200000:2.0:1788572535.345888:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000001862070 17ffffc0001001 3:0 ffff94d408ddb700 348 lru 00000080:00200000:2.0:1788572535.345893:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000f49e2ed3 348: 1< 0 - - > 2< 1425408 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.345895:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000002abf9b14 00000080:00200000:2.0:1788572535.345896:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 93. 00000080:00200000:2.0:1788572535.345898:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 349 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.345910:0:69530:0:(rw26.c:869:ll_write_end()) pos 1429504, len 4096, copied 4096 00000080:00200000:2.0:1788572535.345914:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000f1982487[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 349] 00000080:00200000:2.0:1788572535.345916:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000590f1dd1 17ffffc0001001 3:0 ffff94d408ddb1c0 349 lru 00000080:00200000:2.0:1788572535.345921:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000002d33c17e 349: 1< 0 - - > 2< 1429504 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.345923:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000f1982487 00000080:00200000:2.0:1788572535.345924:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 94. 00000080:00200000:2.0:1788572535.345926:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 350 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.345940:0:69530:0:(rw26.c:869:ll_write_end()) pos 1433600, len 4096, copied 4096 00000080:00200000:2.0:1788572535.345944:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000016f34abc[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 350] 00000080:00200000:2.0:1788572535.345946:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000038086a9c 17ffffc0001001 3:0 ffff94d408ddb620 350 lru 00000080:00200000:2.0:1788572535.345951:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000ca589fe7 350: 1< 0 - - > 2< 1433600 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.345953:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000016f34abc 00000080:00200000:2.0:1788572535.345954:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 95. 00000080:00200000:2.0:1788572535.345956:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 351 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.345970:0:69530:0:(rw26.c:869:ll_write_end()) pos 1437696, len 4096, copied 4096 00000080:00200000:2.0:1788572535.345974:0:69530:0:(rw26.c:927:ll_write_end()) page@000000008cb15613[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 351] 00000080:00200000:2.0:1788572535.345977:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000e02d2df0 17ffffc0001001 3:0 ffff94d408ddbc40 351 lru 00000080:00200000:2.0:1788572535.345982:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000007c1ae61a 351: 1< 0 - - > 2< 1437696 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.345983:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000008cb15613 00000080:00200000:2.0:1788572535.345984:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 96. 00000080:00200000:2.0:1788572535.345988:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 352 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.346009:0:69530:0:(rw26.c:869:ll_write_end()) pos 1441792, len 4096, copied 4096 00000080:00200000:2.0:1788572535.346013:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000e743d2e7[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 352] 00000080:00200000:2.0:1788572535.346016:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000547efa68 17ffffc0001021 2:0 ffff94d408ddb460 352 lru 00000080:00200000:2.0:1788572535.346020:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000023885800 352: 1< 0 - - > 2< 1441792 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.346022:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000e743d2e7 00000080:00200000:2.0:1788572535.346024:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 97. 00000080:00200000:2.0:1788572535.346026:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 353 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.346112:0:69530:0:(rw26.c:869:ll_write_end()) pos 1445888, len 4096, copied 4096 00000080:00200000:2.0:1788572535.346118:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000911fea84[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 353] 00000080:00200000:2.0:1788572535.346120:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000039063b90 17ffffc0001001 3:0 ffff94d408ddb9a0 353 lru 00000080:00200000:2.0:1788572535.346125:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000004c81717f 353: 1< 0 - - > 2< 1445888 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.346127:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000911fea84 00000080:00200000:2.0:1788572535.346128:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 98. 00000080:00200000:2.0:1788572535.346131:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 354 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.346146:0:69530:0:(rw26.c:869:ll_write_end()) pos 1449984, len 4096, copied 4096 00000080:00200000:2.0:1788572535.346150:0:69530:0:(rw26.c:927:ll_write_end()) page@000000004075947e[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 354] 00000080:00200000:2.0:1788572535.346153:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000075f79f89 17ffffc0001001 3:0 ffff94d408ddb8c0 354 lru 00000080:00200000:2.0:1788572535.346159:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000dad14d70 354: 1< 0 - - > 2< 1449984 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.346160:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000004075947e 00000080:00200000:2.0:1788572535.346161:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 99. 00000080:00200000:2.0:1788572535.346164:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 355 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.346176:0:69530:0:(rw26.c:869:ll_write_end()) pos 1454080, len 4096, copied 4096 00000080:00200000:2.0:1788572535.346180:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000ca44380a[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 355] 00000080:00200000:2.0:1788572535.346183:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000e0774e83 17ffffc0001001 3:0 ffff94d408ddbd20 355 lru 00000080:00200000:2.0:1788572535.346188:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000002a6d0ed7 355: 1< 0 - - > 2< 1454080 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.346191:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000ca44380a 00000080:00200000:2.0:1788572535.346192:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 100. 00000080:00200000:2.0:1788572535.346194:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 356 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.346307:0:69530:0:(rw26.c:869:ll_write_end()) pos 1458176, len 4096, copied 4096 00000080:00200000:2.0:1788572535.346703:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000027644882[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 356] 00000080:00200000:2.0:1788572535.346707:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000009dccfa07 17ffffc0001001 3:0 ffff94d408ddb540 356 lru 00000080:00200000:2.0:1788572535.346713:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000007d78be4a 356: 1< 0 - - > 2< 1458176 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.346715:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000027644882 00000080:00200000:2.0:1788572535.346716:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 101. 00000080:00200000:2.0:1788572535.346718:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 357 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.346742:0:69530:0:(rw26.c:869:ll_write_end()) pos 1462272, len 4096, copied 4096 00000080:00200000:2.0:1788572535.346747:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000054279caf[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 357] 00000080:00200000:2.0:1788572535.346749:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000007e0ada6d 17ffffc0001001 3:0 ffff94d408ddb0e0 357 lru 00000080:00200000:2.0:1788572535.346755:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000006a4d3ffe 357: 1< 0 - - > 2< 1462272 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.346757:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000054279caf 00000080:00200000:2.0:1788572535.346758:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 102. 00000080:00200000:2.0:1788572535.346760:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 358 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.346777:0:69530:0:(rw26.c:869:ll_write_end()) pos 1466368, len 4096, copied 4096 00000080:00200000:2.0:1788572535.346781:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000096c6b5ac[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 358] 00000080:00200000:2.0:1788572535.346783:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000007259a606 17ffffc0001001 3:0 ffff94d408ddb380 358 lru 00000080:00200000:2.0:1788572535.346788:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000004bded7d1 358: 1< 0 - - > 2< 1466368 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.346790:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000096c6b5ac 00000080:00200000:2.0:1788572535.346791:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 103. 00000080:00200000:2.0:1788572535.346793:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 359 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.346805:0:69530:0:(rw26.c:869:ll_write_end()) pos 1470464, len 4096, copied 4096 00000080:00200000:2.0:1788572535.346809:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000f6ceafe3[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 359] 00000080:00200000:2.0:1788572535.346812:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000004433185e 17ffffc0001001 3:0 ffff94d408ddb000 359 lru 00000080:00200000:2.0:1788572535.346816:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000007ed9b2e8 359: 1< 0 - - > 2< 1470464 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.346818:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000f6ceafe3 00000080:00200000:2.0:1788572535.346819:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 104. 00000080:00200000:2.0:1788572535.346821:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 360 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.346836:0:69530:0:(rw26.c:869:ll_write_end()) pos 1474560, len 4096, copied 4096 00000080:00200000:2.0:1788572535.346840:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000776c0708[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 360] 00000080:00200000:2.0:1788572535.346842:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000e1af98e1 17ffffc0001001 3:0 ffff94d408ddbee0 360 lru 00000080:00200000:2.0:1788572535.346846:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000002885d672 360: 1< 0 - - > 2< 1474560 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.346849:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000776c0708 00000080:00200000:2.0:1788572535.346850:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 105. 00000080:00200000:2.0:1788572535.346852:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 361 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.346865:0:69530:0:(rw26.c:869:ll_write_end()) pos 1478656, len 4096, copied 4096 00000080:00200000:2.0:1788572535.346870:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000bc137cae[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 361] 00000080:00200000:2.0:1788572535.346873:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000003a0c597d 17ffffc0001001 3:0 ffff94d408ddbe00 361 lru 00000080:00200000:2.0:1788572535.346878:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000413bfc65 361: 1< 0 - - > 2< 1478656 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.346880:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000bc137cae 00000080:00200000:2.0:1788572535.346881:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 106. 00000080:00200000:2.0:1788572535.346883:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 362 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.346896:0:69530:0:(rw26.c:869:ll_write_end()) pos 1482752, len 4096, copied 4096 00000080:00200000:2.0:1788572535.346901:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000caa85fd1[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 362] 00000080:00200000:2.0:1788572535.346904:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000c4b89968 17ffffc0001001 3:0 ffff94d408ddb2a0 362 lru 00000080:00200000:2.0:1788572535.346909:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000e15d3da5 362: 1< 0 - - > 2< 1482752 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.346911:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000caa85fd1 00000080:00200000:2.0:1788572535.346912:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 107. 00000080:00200000:2.0:1788572535.346915:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 363 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.346927:0:69530:0:(rw26.c:869:ll_write_end()) pos 1486848, len 4096, copied 4096 00000080:00200000:2.0:1788572535.346932:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000fb09971f[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 363] 00000080:00200000:2.0:1788572535.346934:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000081a26b49 17ffffc0001001 3:0 ffff94d408d9d540 363 lru 00000080:00200000:2.0:1788572535.346940:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000075ec6ed9 363: 1< 0 - - > 2< 1486848 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.346942:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000fb09971f 00000080:00200000:2.0:1788572535.346943:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 108. 00000080:00200000:2.0:1788572535.346945:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 364 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.346959:0:69530:0:(rw26.c:869:ll_write_end()) pos 1490944, len 4096, copied 4096 00000080:00200000:2.0:1788572535.346964:0:69530:0:(rw26.c:927:ll_write_end()) page@000000003f85ac45[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 364] 00000080:00200000:2.0:1788572535.346967:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000fcdb873d 17ffffc0001001 3:0 ffff94d408d9d0e0 364 lru 00000080:00200000:2.0:1788572535.346972:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000e02f8a53 364: 1< 0 - - > 2< 1490944 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.346974:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000003f85ac45 00000080:00200000:2.0:1788572535.346975:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 109. 00000080:00200000:2.0:1788572535.346977:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 365 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.346994:0:69530:0:(rw26.c:869:ll_write_end()) pos 1495040, len 4096, copied 4096 00000080:00200000:2.0:1788572535.346998:0:69530:0:(rw26.c:927:ll_write_end()) page@000000005aa06af2[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 365] 00000080:00200000:2.0:1788572535.347001:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000593c4964 17ffffc0001001 3:0 ffff94d408d9dee0 365 lru 00000080:00200000:2.0:1788572535.347006:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000d9dbadd3 365: 1< 0 - - > 2< 1495040 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.347008:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000005aa06af2 00000080:00200000:2.0:1788572535.347009:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 110. 00000080:00200000:2.0:1788572535.347011:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 366 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.347026:0:69530:0:(rw26.c:869:ll_write_end()) pos 1499136, len 4096, copied 4096 00000080:00200000:2.0:1788572535.347030:0:69530:0:(rw26.c:927:ll_write_end()) page@000000009ecc6832[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 366] 00000080:00200000:2.0:1788572535.347102:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000026e2f1dc 17ffffc0001001 3:0 ffff94d408d9de00 366 lru 00000080:00200000:2.0:1788572535.347108:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000003ac5ae37 366: 1< 0 - - > 2< 1499136 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.347110:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000009ecc6832 00000080:00200000:2.0:1788572535.347111:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 111. 00000080:00200000:2.0:1788572535.347114:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 367 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.347135:0:69530:0:(rw26.c:869:ll_write_end()) pos 1503232, len 4096, copied 4096 00000080:00200000:2.0:1788572535.347140:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000645165f3[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 367] 00000080:00200000:2.0:1788572535.347143:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000020d359e3 17ffffc0001021 2:0 ffff94d408d9d2a0 367 lru 00000080:00200000:2.0:1788572535.347148:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000008fcdadf1 367: 1< 0 - - > 2< 1503232 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.347152:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000645165f3 00000080:00200000:2.0:1788572535.347153:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 112. 00000080:00200000:2.0:1788572535.347155:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 368 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.347171:0:69530:0:(rw26.c:869:ll_write_end()) pos 1507328, len 4096, copied 4096 00000080:00200000:2.0:1788572535.347175:0:69530:0:(rw26.c:927:ll_write_end()) page@000000005bc940bb[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 368] 00000080:00200000:2.0:1788572535.347178:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000009497edf9 17ffffc0001001 3:0 ffff94d408d9da80 368 lru 00000080:00200000:2.0:1788572535.347182:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000007f1a6137 368: 1< 0 - - > 2< 1507328 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.347184:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000005bc940bb 00000080:00200000:2.0:1788572535.347185:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 113. 00000080:00200000:2.0:1788572535.347187:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 369 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.347446:0:69530:0:(rw26.c:869:ll_write_end()) pos 1511424, len 4096, copied 4096 00000080:00200000:2.0:1788572535.347462:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000092ca3971[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 369] 00000080:00200000:2.0:1788572535.347465:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000609c6f9a 17ffffc0001001 3:0 ffff94d408d9d7e0 369 lru 00000080:00200000:2.0:1788572535.347471:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000d3cb01af 369: 1< 0 - - > 2< 1511424 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.347473:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000092ca3971 00000080:00200000:2.0:1788572535.347474:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 114. 00000080:00200000:2.0:1788572535.347476:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 370 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.347493:0:69530:0:(rw26.c:869:ll_write_end()) pos 1515520, len 4096, copied 4096 00000080:00200000:2.0:1788572535.347498:0:69530:0:(rw26.c:927:ll_write_end()) page@000000004a653629[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 370] 00000080:00200000:2.0:1788572535.347501:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000f667079f 17ffffc0001001 3:0 ffff94d408d9d700 370 lru 00000080:00200000:2.0:1788572535.347505:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000368ee48f 370: 1< 0 - - > 2< 1515520 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.347508:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000004a653629 00000080:00200000:2.0:1788572535.347509:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 115. 00000080:00200000:2.0:1788572535.347511:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 371 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.347525:0:69530:0:(rw26.c:869:ll_write_end()) pos 1519616, len 4096, copied 4096 00000080:00200000:2.0:1788572535.347530:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000e2a3de47[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 371] 00000080:00200000:2.0:1788572535.347533:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000ecc233f4 17ffffc0001001 3:0 ffff94d408d9d1c0 371 lru 00000080:00200000:2.0:1788572535.347539:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000008cac6da2 371: 1< 0 - - > 2< 1519616 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.347541:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000e2a3de47 00000080:00200000:2.0:1788572535.347542:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 116. 00000080:00200000:2.0:1788572535.347544:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 372 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.347565:0:69530:0:(rw26.c:869:ll_write_end()) pos 1523712, len 4096, copied 4096 00000080:00200000:2.0:1788572535.347569:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000669717d6[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 372] 00000080:00200000:2.0:1788572535.347571:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000a4889115 17ffffc0001001 3:0 ffff94d408d9dc40 372 lru 00000080:00200000:2.0:1788572535.347576:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000005d1f01b0 372: 1< 0 - - > 2< 1523712 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.347579:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000669717d6 00000080:00200000:2.0:1788572535.347580:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 117. 00000080:00200000:2.0:1788572535.347581:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 373 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.347593:0:69530:0:(rw26.c:869:ll_write_end()) pos 1527808, len 4096, copied 4096 00000080:00200000:2.0:1788572535.347597:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000c55e0ca7[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 373] 00000080:00200000:2.0:1788572535.347599:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000006f3f30ad 17ffffc0001001 3:0 ffff94d408d9d460 373 lru 00000080:00200000:2.0:1788572535.347604:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000b8e41b80 373: 1< 0 - - > 2< 1527808 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.347605:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000c55e0ca7 00000080:00200000:2.0:1788572535.347606:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 118. 00000080:00200000:2.0:1788572535.347608:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 374 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.347623:0:69530:0:(rw26.c:869:ll_write_end()) pos 1531904, len 4096, copied 4096 00000080:00200000:2.0:1788572535.347627:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000eff3b649[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 374] 00000080:00200000:2.0:1788572535.347630:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000003f95cae5 17ffffc0001001 3:0 ffff94d408d9db60 374 lru 00000080:00200000:2.0:1788572535.347635:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000032c42bf9 374: 1< 0 - - > 2< 1531904 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.347637:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000eff3b649 00000080:00200000:2.0:1788572535.347638:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 119. 00000080:00200000:2.0:1788572535.347640:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 375 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.347653:0:69530:0:(rw26.c:869:ll_write_end()) pos 1536000, len 4096, copied 4096 00000080:00200000:2.0:1788572535.347658:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000026d75e8[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 375] 00000080:00200000:2.0:1788572535.347660:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000ebf30eab 17ffffc0001001 3:0 ffff94d408d9d620 375 lru 00000080:00200000:2.0:1788572535.347666:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000004c4670d4 375: 1< 0 - - > 2< 1536000 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.347668:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000026d75e8 00000080:00200000:2.0:1788572535.347669:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 120. 00000080:00200000:2.0:1788572535.347670:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 376 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.347686:0:69530:0:(rw26.c:869:ll_write_end()) pos 1540096, len 4096, copied 4096 00000080:00200000:2.0:1788572535.347690:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000144cecde[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 376] 00000080:00200000:2.0:1788572535.347692:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000a2a2b7f2 17ffffc0001001 3:0 ffff94d408d9d8c0 376 lru 00000080:00200000:2.0:1788572535.347698:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000008b543fe2 376: 1< 0 - - > 2< 1540096 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.347700:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000144cecde 00000080:00200000:2.0:1788572535.347701:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 121. 00000080:00200000:2.0:1788572535.347703:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 377 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.347716:0:69530:0:(rw26.c:869:ll_write_end()) pos 1544192, len 4096, copied 4096 00000080:00200000:2.0:1788572535.347720:0:69530:0:(rw26.c:927:ll_write_end()) page@000000002959abf6[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 377] 00000080:00200000:2.0:1788572535.347723:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000005e5abcdd 17ffffc0001001 3:0 ffff94d408d9d9a0 377 lru 00000080:00200000:2.0:1788572535.347727:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000438cf667 377: 1< 0 - - > 2< 1544192 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.347729:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000002959abf6 00000080:00200000:2.0:1788572535.347730:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 122. 00000080:00200000:2.0:1788572535.347732:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 378 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.347744:0:69530:0:(rw26.c:869:ll_write_end()) pos 1548288, len 4096, copied 4096 00000080:00200000:2.0:1788572535.347748:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000e5990f32[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 378] 00000080:00200000:2.0:1788572535.347751:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000d26f2c1b 17ffffc0001001 3:0 ffff94d408d9dd20 378 lru 00000080:00200000:2.0:1788572535.347755:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000b86f18a7 378: 1< 0 - - > 2< 1548288 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.347757:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000e5990f32 00000080:00200000:2.0:1788572535.347758:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 123. 00000080:00200000:2.0:1788572535.347759:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 379 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.347774:0:69530:0:(rw26.c:869:ll_write_end()) pos 1552384, len 4096, copied 4096 00000080:00200000:2.0:1788572535.347778:0:69530:0:(rw26.c:927:ll_write_end()) page@000000009dd34eb6[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 379] 00000080:00200000:2.0:1788572535.347781:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000094a9a8cd 17ffffc0001001 3:0 ffff94d408d9d380 379 lru 00000080:00200000:2.0:1788572535.347786:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000009d5cb7c1 379: 1< 0 - - > 2< 1552384 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.347788:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000009dd34eb6 00000080:00200000:2.0:1788572535.347789:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 124. 00000080:00200000:2.0:1788572535.347791:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 380 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.347805:0:69530:0:(rw26.c:869:ll_write_end()) pos 1556480, len 4096, copied 4096 00000080:00200000:2.0:1788572535.347810:0:69530:0:(rw26.c:927:ll_write_end()) page@000000006a077631[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 380] 00000080:00200000:2.0:1788572535.347813:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000df941844 17ffffc0001001 3:0 ffff94d408d9d000 380 lru 00000080:00200000:2.0:1788572535.347818:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000ed0978a8 380: 1< 0 - - > 2< 1556480 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.347820:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000006a077631 00000080:00200000:2.0:1788572535.347821:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 125. 00000080:00200000:2.0:1788572535.347823:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 381 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.347838:0:69530:0:(rw26.c:869:ll_write_end()) pos 1560576, len 4096, copied 4096 00000080:00200000:2.0:1788572535.347843:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000bba38da4[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 381] 00000080:00200000:2.0:1788572535.347845:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000c7506221 17ffffc0001001 3:0 ffff94d408dbd700 381 lru 00000080:00200000:2.0:1788572535.347850:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000b7e24ef6 381: 1< 0 - - > 2< 1560576 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.347852:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000bba38da4 00000080:00200000:2.0:1788572535.347853:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 126. 00000080:00200000:2.0:1788572535.347855:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 382 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.347875:0:69530:0:(rw26.c:869:ll_write_end()) pos 1564672, len 4096, copied 4096 00000080:00200000:2.0:1788572535.347879:0:69530:0:(rw26.c:927:ll_write_end()) page@000000002a6930fd[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 382] 00000080:00200000:2.0:1788572535.347882:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000bc83140d 17ffffc0001021 2:0 ffff94d408dbd1c0 382 lru 00000080:00200000:2.0:1788572535.347886:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000000baddd3d 382: 1< 0 - - > 2< 1564672 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.347887:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000002a6930fd 00000080:00200000:2.0:1788572535.347888:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 127. 00000080:00200000:2.0:1788572535.347890:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 383 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.348125:0:69530:0:(rw26.c:869:ll_write_end()) pos 1568768, len 4096, copied 4096 00000080:00200000:2.0:1788572535.348131:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000008ec590a[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 383] 00000080:00200000:2.0:1788572535.348134:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000451e8a00 17ffffc0001001 3:0 ffff94d408dbdc40 383 lru 00000080:00200000:2.0:1788572535.348139:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000506411b3 383: 1< 0 - - > 2< 1568768 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.348141:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000008ec590a 00000080:00200000:2.0:1788572535.348142:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 128. 00000080:00200000:2.0:1788572535.348145:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 384 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.348166:0:69530:0:(rw26.c:869:ll_write_end()) pos 1572864, len 4096, copied 4096 00000080:00200000:2.0:1788572535.348170:0:69530:0:(rw26.c:927:ll_write_end()) page@000000000b007521[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 384] 00000080:00200000:2.0:1788572535.348173:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000007884c264 17ffffc0001001 3:0 ffff94d408dbd380 384 lru 00000080:00200000:2.0:1788572535.348178:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000003b12af41 384: 1< 0 - - > 2< 1572864 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.348180:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000000b007521 00000080:00200000:2.0:1788572535.348181:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 129. 00000080:00200000:2.0:1788572535.348183:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 385 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.348197:0:69530:0:(rw26.c:869:ll_write_end()) pos 1576960, len 4096, copied 4096 00000080:00200000:2.0:1788572535.348303:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000063f94bde[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 385] 00000080:00200000:2.0:1788572535.348307:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000f80264ce 17ffffc0001001 3:0 ffff94d408dbd000 385 lru 00000080:00200000:2.0:1788572535.348379:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000716235c5 385: 1< 0 - - > 2< 1576960 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.348382:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000063f94bde 00000080:00200000:2.0:1788572535.348384:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 130. 00000080:00200000:2.0:1788572535.348386:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 386 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.348406:0:69530:0:(rw26.c:869:ll_write_end()) pos 1581056, len 4096, copied 4096 00000080:00200000:2.0:1788572535.348411:0:69530:0:(rw26.c:927:ll_write_end()) page@000000008356add8[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 386] 00000080:00200000:2.0:1788572535.348413:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000054d128f5 17ffffc0001001 3:0 ffff94d408dbdee0 386 lru 00000080:00200000:2.0:1788572535.348418:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000024c191f5 386: 1< 0 - - > 2< 1581056 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.348420:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000008356add8 00000080:00200000:2.0:1788572535.348421:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 131. 00000080:00200000:2.0:1788572535.348423:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 387 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.348439:0:69530:0:(rw26.c:869:ll_write_end()) pos 1585152, len 4096, copied 4096 00000080:00200000:2.0:1788572535.348444:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000093dd19c9[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 387] 00000080:00200000:2.0:1788572535.348447:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000c61b3ca6 17ffffc0001001 3:0 ffff94d408dbd9a0 387 lru 00000080:00200000:2.0:1788572535.348461:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000d927c62e 387: 1< 0 - - > 2< 1585152 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.348463:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000093dd19c9 00000080:00200000:2.0:1788572535.348464:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 132. 00000080:00200000:2.0:1788572535.348466:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 388 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.348482:0:69530:0:(rw26.c:869:ll_write_end()) pos 1589248, len 4096, copied 4096 00000080:00200000:2.0:1788572535.348487:0:69530:0:(rw26.c:927:ll_write_end()) page@000000006e6432d2[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 388] 00000080:00200000:2.0:1788572535.348490:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000e4d6f8fd 17ffffc0001001 3:0 ffff94d408dbdd20 388 lru 00000080:00200000:2.0:1788572535.348495:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000083021b66 388: 1< 0 - - > 2< 1589248 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.348496:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000006e6432d2 00000080:00200000:2.0:1788572535.348498:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 133. 00000080:00200000:2.0:1788572535.348500:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 389 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.348514:0:69530:0:(rw26.c:869:ll_write_end()) pos 1593344, len 4096, copied 4096 00000080:00200000:2.0:1788572535.348519:0:69530:0:(rw26.c:927:ll_write_end()) page@000000000aa0491b[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 389] 00000080:00200000:2.0:1788572535.348521:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000ec0c819c 17ffffc0001001 3:0 ffff94d408dbd460 389 lru 00000080:00200000:2.0:1788572535.348525:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000b18a1d01 389: 1< 0 - - > 2< 1593344 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.348529:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000000aa0491b 00000080:00200000:2.0:1788572535.348530:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 134. 00000080:00200000:2.0:1788572535.348531:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 390 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.348543:0:69530:0:(rw26.c:869:ll_write_end()) pos 1597440, len 4096, copied 4096 00000080:00200000:2.0:1788572535.348547:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000d8dd64f1[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 390] 00000080:00200000:2.0:1788572535.348549:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000023a8af93 17ffffc0001001 3:0 ffff94d408dbd8c0 390 lru 00000080:00200000:2.0:1788572535.348553:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000043caae22 390: 1< 0 - - > 2< 1597440 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.348555:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000d8dd64f1 00000080:00200000:2.0:1788572535.348556:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 135. 00000080:00200000:2.0:1788572535.348557:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 391 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.348570:0:69530:0:(rw26.c:869:ll_write_end()) pos 1601536, len 4096, copied 4096 00000080:00200000:2.0:1788572535.348573:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000067d5775a[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 391] 00000080:00200000:2.0:1788572535.348576:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000ff8fcd31 17ffffc0001001 3:0 ffff94d408dbde00 391 lru 00000080:00200000:2.0:1788572535.348580:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000009ffc801e 391: 1< 0 - - > 2< 1601536 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.348582:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000067d5775a 00000080:00200000:2.0:1788572535.348583:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 136. 00000080:00200000:2.0:1788572535.348584:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 392 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.348598:0:69530:0:(rw26.c:869:ll_write_end()) pos 1605632, len 4096, copied 4096 00000080:00200000:2.0:1788572535.348602:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000058d11dbd[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 392] 00000080:00200000:2.0:1788572535.348604:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000adfbcecf 17ffffc0001001 3:0 ffff94d408dbd0e0 392 lru 00000080:00200000:2.0:1788572535.348609:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000953f735f 392: 1< 0 - - > 2< 1605632 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.348611:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000058d11dbd 00000080:00200000:2.0:1788572535.348612:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 137. 00000080:00200000:2.0:1788572535.348614:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 393 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.348629:0:69530:0:(rw26.c:869:ll_write_end()) pos 1609728, len 4096, copied 4096 00000080:00200000:2.0:1788572535.348634:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000e30f4c6a[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 393] 00000080:00200000:2.0:1788572535.348636:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000004829dd23 17ffffc0001001 3:0 ffff94d408dbd540 393 lru 00000080:00200000:2.0:1788572535.348640:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000db9d5d74 393: 1< 0 - - > 2< 1609728 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.348642:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000e30f4c6a 00000080:00200000:2.0:1788572535.348643:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 138. 00000080:00200000:2.0:1788572535.348646:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 394 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.348663:0:69530:0:(rw26.c:869:ll_write_end()) pos 1613824, len 4096, copied 4096 00000080:00200000:2.0:1788572535.348667:0:69530:0:(rw26.c:927:ll_write_end()) page@000000008a22c424[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 394] 00000080:00200000:2.0:1788572535.348669:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000b92b174a 17ffffc0001001 3:0 ffff94d408dbdb60 394 lru 00000080:00200000:2.0:1788572535.348675:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000007d946214 394: 1< 0 - - > 2< 1613824 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.348677:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000008a22c424 00000080:00200000:2.0:1788572535.348678:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 139. 00000080:00200000:2.0:1788572535.348679:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 395 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.348694:0:69530:0:(rw26.c:869:ll_write_end()) pos 1617920, len 4096, copied 4096 00000080:00200000:2.0:1788572535.348698:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000e8d5ed17[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 395] 00000080:00200000:2.0:1788572535.348701:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000004579a64e 17ffffc0001001 3:0 ffff94d408dbd620 395 lru 00000080:00200000:2.0:1788572535.348706:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000006eba34a6 395: 1< 0 - - > 2< 1617920 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.348708:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000e8d5ed17 00000080:00200000:2.0:1788572535.348709:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 140. 00000080:00200000:2.0:1788572535.348711:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 396 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.348725:0:69530:0:(rw26.c:869:ll_write_end()) pos 1622016, len 4096, copied 4096 00000080:00200000:2.0:1788572535.348728:0:69530:0:(rw26.c:927:ll_write_end()) page@000000003eba5793[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 396] 00000080:00200000:2.0:1788572535.348731:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000fe577ff0 17ffffc0001001 3:0 ffff94d408dbd2a0 396 lru 00000080:00200000:2.0:1788572535.348735:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000008bcf8015 396: 1< 0 - - > 2< 1622016 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.348737:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000003eba5793 00000080:00200000:2.0:1788572535.348738:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 141. 00000080:00200000:2.0:1788572535.348739:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 397 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.348756:0:69530:0:(rw26.c:869:ll_write_end()) pos 1626112, len 4096, copied 4096 00000080:00200000:2.0:1788572535.348759:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000a37086a8[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 397] 00000080:00200000:2.0:1788572535.348762:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000001b48f8f6 17ffffc0001021 2:0 ffff94d408dbd7e0 397 lru 00000080:00200000:2.0:1788572535.348768:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000001862494a 397: 1< 0 - - > 2< 1626112 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.348770:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000a37086a8 00000080:00200000:2.0:1788572535.348771:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 142. 00000080:00200000:2.0:1788572535.348773:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 398 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.348789:0:69530:0:(rw26.c:869:ll_write_end()) pos 1630208, len 4096, copied 4096 00000080:00200000:2.0:1788572535.348793:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000a112b005[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 398] 00000080:00200000:2.0:1788572535.348795:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000aaae26e7 17ffffc0001001 3:0 ffff94d408dbda80 398 lru 00000080:00200000:2.0:1788572535.348801:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000f82eba0b 398: 1< 0 - - > 2< 1630208 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.348803:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000a112b005 00000080:00200000:2.0:1788572535.348804:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 143. 00000080:00200000:2.0:1788572535.348806:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 399 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.348820:0:69530:0:(rw26.c:869:ll_write_end()) pos 1634304, len 4096, copied 4096 00000080:00200000:2.0:1788572535.348824:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000032574940[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 399] 00000080:00200000:2.0:1788572535.348827:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000036536524 17ffffc0001001 3:0 ffff94d416f3f9a0 399 lru 00000080:00200000:2.0:1788572535.348832:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000075cf7bee 399: 1< 0 - - > 2< 1634304 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.348835:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000032574940 00000080:00200000:2.0:1788572535.348836:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 144. 00000080:00200000:2.0:1788572535.348838:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 400 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.348855:0:69530:0:(rw26.c:869:ll_write_end()) pos 1638400, len 4096, copied 4096 00000080:00200000:2.0:1788572535.348859:0:69530:0:(rw26.c:927:ll_write_end()) page@000000008d7fe69f[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 400] 00000080:00200000:2.0:1788572535.348861:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000a0da90fb 17ffffc0001001 3:0 ffff94d416f3f1c0 400 lru 00000080:00200000:2.0:1788572535.348866:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000004fc4646d 400: 1< 0 - - > 2< 1638400 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.348868:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000008d7fe69f 00000080:00200000:2.0:1788572535.348869:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 145. 00000080:00200000:2.0:1788572535.348871:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 401 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.348886:0:69530:0:(rw26.c:869:ll_write_end()) pos 1642496, len 4096, copied 4096 00000080:00200000:2.0:1788572535.348891:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000fd84582f[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 401] 00000080:00200000:2.0:1788572535.348894:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000c9606ff6 17ffffc0001001 3:0 ffff94d416f3fc40 401 lru 00000080:00200000:2.0:1788572535.349543:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000004457af41 401: 1< 0 - - > 2< 1642496 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.349546:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000fd84582f 00000080:00200000:2.0:1788572535.349547:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 146. 00000080:00200000:2.0:1788572535.349550:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 402 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.349575:0:69530:0:(rw26.c:869:ll_write_end()) pos 1646592, len 4096, copied 4096 00000080:00200000:2.0:1788572535.349581:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000083f4cded[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 402] 00000080:00200000:2.0:1788572535.349584:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000004d29f594 17ffffc0001001 3:0 ffff94d416f3fd20 402 lru 00000080:00200000:2.0:1788572535.349588:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000005ffdf49e 402: 1< 0 - - > 2< 1646592 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.349590:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000083f4cded 00000080:00200000:2.0:1788572535.349592:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 147. 00000080:00200000:2.0:1788572535.349593:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 403 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.349608:0:69530:0:(rw26.c:869:ll_write_end()) pos 1650688, len 4096, copied 4096 00000080:00200000:2.0:1788572535.349613:0:69530:0:(rw26.c:927:ll_write_end()) page@000000007257a6df[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 403] 00000080:00200000:2.0:1788572535.349615:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000000e575c5a 17ffffc0001001 3:0 ffff94d416f3f460 403 lru 00000080:00200000:2.0:1788572535.349620:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000e0ccfae0 403: 1< 0 - - > 2< 1650688 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.349622:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000007257a6df 00000080:00200000:2.0:1788572535.349623:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 148. 00000080:00200000:2.0:1788572535.349625:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 404 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.349639:0:69530:0:(rw26.c:869:ll_write_end()) pos 1654784, len 4096, copied 4096 00000080:00200000:2.0:1788572535.349644:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000033664bd[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 404] 00000080:00200000:2.0:1788572535.349647:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000c5c58324 17ffffc0001001 3:0 ffff94d416f3f8c0 404 lru 00000080:00200000:2.0:1788572535.349653:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000001e05b28c 404: 1< 0 - - > 2< 1654784 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.349655:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000033664bd 00000080:00200000:2.0:1788572535.349656:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 149. 00000080:00200000:2.0:1788572535.349658:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 405 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.349672:0:69530:0:(rw26.c:869:ll_write_end()) pos 1658880, len 4096, copied 4096 00000080:00200000:2.0:1788572535.349677:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000da24f2a1[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 405] 00000080:00200000:2.0:1788572535.349680:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000335c0f64 17ffffc0001001 3:0 ffff94d416f3f380 405 lru 00000080:00200000:2.0:1788572535.349685:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000b28415b3 405: 1< 0 - - > 2< 1658880 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.349688:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000da24f2a1 00000080:00200000:2.0:1788572535.349689:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 150. 00000080:00200000:2.0:1788572535.349691:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 406 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.349705:0:69530:0:(rw26.c:869:ll_write_end()) pos 1662976, len 4096, copied 4096 00000080:00200000:2.0:1788572535.349710:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000002f504bb[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 406] 00000080:00200000:2.0:1788572535.349713:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000001be2116 17ffffc0001001 3:0 ffff94d416f3f000 406 lru 00000080:00200000:2.0:1788572535.349719:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000042c2745f 406: 1< 0 - - > 2< 1662976 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.349722:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000002f504bb 00000080:00200000:2.0:1788572535.349723:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 151. 00000080:00200000:2.0:1788572535.349725:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 407 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.349740:0:69530:0:(rw26.c:869:ll_write_end()) pos 1667072, len 4096, copied 4096 00000080:00200000:2.0:1788572535.349744:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000927ac85b[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 407] 00000080:00200000:2.0:1788572535.349747:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000051a9fbd5 17ffffc0001001 3:0 ffff94d416f3fee0 407 lru 00000080:00200000:2.0:1788572535.349752:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000d99ce377 407: 1< 0 - - > 2< 1667072 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.349754:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000927ac85b 00000080:00200000:2.0:1788572535.349755:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 152. 00000080:00200000:2.0:1788572535.349757:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 408 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.349772:0:69530:0:(rw26.c:869:ll_write_end()) pos 1671168, len 4096, copied 4096 00000080:00200000:2.0:1788572535.349777:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000b8d0506f[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 408] 00000080:00200000:2.0:1788572535.349779:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000003fd4165d 17ffffc0001001 3:0 ffff94d416f3f540 408 lru 00000080:00200000:2.0:1788572535.349784:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000613d354a 408: 1< 0 - - > 2< 1671168 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.349786:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000b8d0506f 00000080:00200000:2.0:1788572535.349787:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 153. 00000080:00200000:2.0:1788572535.349790:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 409 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.349805:0:69530:0:(rw26.c:869:ll_write_end()) pos 1675264, len 4096, copied 4096 00000080:00200000:2.0:1788572535.349809:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000b4dcbb67[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 409] 00000080:00200000:2.0:1788572535.349812:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000ed4898a6 17ffffc0001001 3:0 ffff94d416f3fe00 409 lru 00000080:00200000:2.0:1788572535.349818:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000be976e2b 409: 1< 0 - - > 2< 1675264 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.349820:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000b4dcbb67 00000080:00200000:2.0:1788572535.349821:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 154. 00000080:00200000:2.0:1788572535.349823:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 410 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.349838:0:69530:0:(rw26.c:869:ll_write_end()) pos 1679360, len 4096, copied 4096 00000080:00200000:2.0:1788572535.349842:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000030c524fa[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 410] 00000080:00200000:2.0:1788572535.349845:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000e0ba51f9 17ffffc0001001 3:0 ffff94d416f3f2a0 410 lru 00000080:00200000:2.0:1788572535.349850:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000f70176fd 410: 1< 0 - - > 2< 1679360 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.349851:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000030c524fa 00000080:00200000:2.0:1788572535.349852:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 155. 00000080:00200000:2.0:1788572535.349854:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 411 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.349868:0:69530:0:(rw26.c:869:ll_write_end()) pos 1683456, len 4096, copied 4096 00000080:00200000:2.0:1788572535.349872:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000e1ce8c2d[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 411] 00000080:00200000:2.0:1788572535.349875:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000428ad37a 17ffffc0001001 3:0 ffff94d416f3f0e0 411 lru 00000080:00200000:2.0:1788572535.349880:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000001e3547a0 411: 1< 0 - - > 2< 1683456 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.349885:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000e1ce8c2d 00000080:00200000:2.0:1788572535.349886:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 156. 00000080:00200000:2.0:1788572535.349888:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 412 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.349980:0:69530:0:(rw26.c:869:ll_write_end()) pos 1687552, len 4096, copied 4096 00000080:00200000:2.0:1788572535.349985:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000c40913cd[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 412] 00000080:00200000:2.0:1788572535.349988:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000074870b15 17ffffc0001021 2:0 ffff94d416f3f7e0 412 lru 00000080:00200000:2.0:1788572535.349994:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000053156b85 412: 1< 0 - - > 2< 1687552 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.349996:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000c40913cd 00000080:00200000:2.0:1788572535.349997:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 157. 00000080:00200000:2.0:1788572535.349999:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 413 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.350015:0:69530:0:(rw26.c:869:ll_write_end()) pos 1691648, len 4096, copied 4096 00000080:00200000:2.0:1788572535.350019:0:69530:0:(rw26.c:927:ll_write_end()) page@000000004858114a[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 413] 00000080:00200000:2.0:1788572535.350022:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000c76560dc 17ffffc0001001 3:0 ffff94d416f3fb60 413 lru 00000080:00200000:2.0:1788572535.350028:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000075898432 413: 1< 0 - - > 2< 1691648 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.350096:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000004858114a 00000080:00200000:2.0:1788572535.350099:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 158. 00000080:00200000:2.0:1788572535.350101:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 414 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.350120:0:69530:0:(rw26.c:869:ll_write_end()) pos 1695744, len 4096, copied 4096 00000080:00200000:2.0:1788572535.350125:0:69530:0:(rw26.c:927:ll_write_end()) page@000000004f35f844[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 414] 00000080:00200000:2.0:1788572535.350128:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000e79b8088 17ffffc0001001 3:0 ffff94d416f3fa80 414 lru 00000080:00200000:2.0:1788572535.350134:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000a8b2c86f 414: 1< 0 - - > 2< 1695744 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.350137:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000004f35f844 00000080:00200000:2.0:1788572535.350138:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 159. 00000080:00200000:2.0:1788572535.350139:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 415 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.350151:0:69530:0:(rw26.c:869:ll_write_end()) pos 1699840, len 4096, copied 4096 00000080:00200000:2.0:1788572535.350155:0:69530:0:(rw26.c:927:ll_write_end()) page@000000000313a500[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 415] 00000080:00200000:2.0:1788572535.350159:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000dbfa7fe2 17ffffc0001001 3:0 ffff94d416f3f620 415 lru 00000080:00200000:2.0:1788572535.350163:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000092a8e642 415: 1< 0 - - > 2< 1699840 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.350166:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000000313a500 00000080:00200000:2.0:1788572535.350167:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 160. 00000080:00200000:2.0:1788572535.350170:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 416 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.350183:0:69530:0:(rw26.c:869:ll_write_end()) pos 1703936, len 4096, copied 4096 00000080:00200000:2.0:1788572535.350187:0:69530:0:(rw26.c:927:ll_write_end()) page@000000007b34f65b[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 416] 00000080:00200000:2.0:1788572535.350190:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000006e77a61d 17ffffc0001001 3:0 ffff94d416f3f700 416 lru 00000080:00200000:2.0:1788572535.350195:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000087c1e11e 416: 1< 0 - - > 2< 1703936 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.350198:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000007b34f65b 00000080:00200000:2.0:1788572535.350199:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 161. 00000080:00200000:2.0:1788572535.350298:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 417 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.350392:0:69530:0:(rw26.c:869:ll_write_end()) pos 1708032, len 4096, copied 4096 00000080:00200000:2.0:1788572535.350398:0:69530:0:(rw26.c:927:ll_write_end()) page@000000001807a881[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 417] 00000080:00200000:2.0:1788572535.350401:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000a3858a83 17ffffc0001001 3:0 ffff94d41257bd20 417 lru 00000080:00200000:2.0:1788572535.350407:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000a6e1ee93 417: 1< 0 - - > 2< 1708032 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.350410:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000001807a881 00000080:00200000:2.0:1788572535.350411:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 162. 00000080:00200000:2.0:1788572535.350413:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 418 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.350428:0:69530:0:(rw26.c:869:ll_write_end()) pos 1712128, len 4096, copied 4096 00000080:00200000:2.0:1788572535.350433:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000b04fa333[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 418] 00000080:00200000:2.0:1788572535.350436:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000e5125aca 17ffffc0001001 3:0 ffff94d41257bc40 418 lru 00000080:00200000:2.0:1788572535.350441:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000e422a556 418: 1< 0 - - > 2< 1712128 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.350444:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000b04fa333 00000080:00200000:2.0:1788572535.350445:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 163. 00000080:00200000:2.0:1788572535.350447:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 419 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.350471:0:69530:0:(rw26.c:869:ll_write_end()) pos 1716224, len 4096, copied 4096 00000080:00200000:2.0:1788572535.350476:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000041445de5[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 419] 00000080:00200000:2.0:1788572535.350479:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000d847afe7 17ffffc0001001 3:0 ffff94d41257b380 419 lru 00000080:00200000:2.0:1788572535.350484:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000aa599a34 419: 1< 0 - - > 2< 1716224 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.350486:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000041445de5 00000080:00200000:2.0:1788572535.350487:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 164. 00000080:00200000:2.0:1788572535.350489:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 420 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.350507:0:69530:0:(rw26.c:869:ll_write_end()) pos 1720320, len 4096, copied 4096 00000080:00200000:2.0:1788572535.350511:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000f797b071[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 420] 00000080:00200000:2.0:1788572535.350514:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000c57e79b9 17ffffc0001001 3:0 ffff94d41257b460 420 lru 00000080:00200000:2.0:1788572535.350519:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000ca174df5 420: 1< 0 - - > 2< 1720320 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.350520:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000f797b071 00000080:00200000:2.0:1788572535.350521:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 165. 00000080:00200000:2.0:1788572535.350523:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 421 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.350542:0:69530:0:(rw26.c:869:ll_write_end()) pos 1724416, len 4096, copied 4096 00000080:00200000:2.0:1788572535.350546:0:69530:0:(rw26.c:927:ll_write_end()) page@000000003248f947[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 421] 00000080:00200000:2.0:1788572535.350548:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000001c622b81 17ffffc0001001 3:0 ffff94d41257b000 421 lru 00000080:00200000:2.0:1788572535.350553:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000d883bed1 421: 1< 0 - - > 2< 1724416 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.350555:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000003248f947 00000080:00200000:2.0:1788572535.350556:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 166. 00000080:00200000:2.0:1788572535.350558:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 422 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.350571:0:69530:0:(rw26.c:869:ll_write_end()) pos 1728512, len 4096, copied 4096 00000080:00200000:2.0:1788572535.350574:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000aebbfbaa[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 422] 00000080:00200000:2.0:1788572535.350577:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000004953543c 17ffffc0001001 3:0 ffff94d41257b8c0 422 lru 00000080:00200000:2.0:1788572535.350581:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000005596247 422: 1< 0 - - > 2< 1728512 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.350583:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000aebbfbaa 00000080:00200000:2.0:1788572535.350584:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 167. 00000080:00200000:2.0:1788572535.350586:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 423 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.350602:0:69530:0:(rw26.c:869:ll_write_end()) pos 1732608, len 4096, copied 4096 00000080:00200000:2.0:1788572535.350607:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000b2e67d55[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 423] 00000080:00200000:2.0:1788572535.350609:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000606114b2 17ffffc0001001 3:0 ffff94d41257bee0 423 lru 00000080:00200000:2.0:1788572535.350613:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000e861aeb6 423: 1< 0 - - > 2< 1732608 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.350615:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000b2e67d55 00000080:00200000:2.0:1788572535.350616:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 168. 00000080:00200000:2.0:1788572535.350618:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 424 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.350629:0:69530:0:(rw26.c:869:ll_write_end()) pos 1736704, len 4096, copied 4096 00000080:00200000:2.0:1788572535.350634:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000061f42b7c[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 424] 00000080:00200000:2.0:1788572535.350636:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000003e437301 17ffffc0001001 3:0 ffff94d41257b540 424 lru 00000080:00200000:2.0:1788572535.350641:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000dac235bc 424: 1< 0 - - > 2< 1736704 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.350643:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000061f42b7c 00000080:00200000:2.0:1788572535.350644:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 169. 00000080:00200000:2.0:1788572535.350645:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 425 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.350657:0:69530:0:(rw26.c:869:ll_write_end()) pos 1740800, len 4096, copied 4096 00000080:00200000:2.0:1788572535.350661:0:69530:0:(rw26.c:927:ll_write_end()) page@000000006a698447[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 425] 00000080:00200000:2.0:1788572535.350663:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000014dee68a 17ffffc0001001 3:0 ffff94d41257be00 425 lru 00000080:00200000:2.0:1788572535.350667:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000777723d0 425: 1< 0 - - > 2< 1740800 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.350669:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000006a698447 00000080:00200000:2.0:1788572535.350671:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 170. 00000080:00200000:2.0:1788572535.350672:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 426 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.350685:0:69530:0:(rw26.c:869:ll_write_end()) pos 1744896, len 4096, copied 4096 00000080:00200000:2.0:1788572535.350690:0:69530:0:(rw26.c:927:ll_write_end()) page@000000007245d325[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 426] 00000080:00200000:2.0:1788572535.350692:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000079d1dac7 17ffffc0001001 3:0 ffff94d41257b0e0 426 lru 00000080:00200000:2.0:1788572535.350697:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000079c50843 426: 1< 0 - - > 2< 1744896 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.350699:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000007245d325 00000080:00200000:2.0:1788572535.350700:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 171. 00000080:00200000:2.0:1788572535.350702:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 427 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.350720:0:69530:0:(rw26.c:869:ll_write_end()) pos 1748992, len 4096, copied 4096 00000080:00200000:2.0:1788572535.350724:0:69530:0:(rw26.c:927:ll_write_end()) page@000000003f274fa0[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 427] 00000080:00200000:2.0:1788572535.350727:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000005a0d89c1 17ffffc0001021 2:0 ffff94d41257b2a0 427 lru 00000080:00200000:2.0:1788572535.350732:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000df82c88c 427: 1< 0 - - > 2< 1748992 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.350733:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000003f274fa0 00000080:00200000:2.0:1788572535.350734:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 172. 00000080:00200000:2.0:1788572535.350736:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 428 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.350753:0:69530:0:(rw26.c:869:ll_write_end()) pos 1753088, len 4096, copied 4096 00000080:00200000:2.0:1788572535.350757:0:69530:0:(rw26.c:927:ll_write_end()) page@000000003087f8e5[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 428] 00000080:00200000:2.0:1788572535.350759:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000009aeb1737 17ffffc0001001 3:0 ffff94d41257b7e0 428 lru 00000080:00200000:2.0:1788572535.350765:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000005434172a 428: 1< 0 - - > 2< 1753088 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.350767:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000003087f8e5 00000080:00200000:2.0:1788572535.350768:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 173. 00000080:00200000:2.0:1788572535.350770:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 429 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.350785:0:69530:0:(rw26.c:869:ll_write_end()) pos 1757184, len 4096, copied 4096 00000080:00200000:2.0:1788572535.350789:0:69530:0:(rw26.c:927:ll_write_end()) page@000000000b3b738c[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 429] 00000080:00200000:2.0:1788572535.350792:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000002caa7aa0 17ffffc0001001 3:0 ffff94d41257bb60 429 lru 00000080:00200000:2.0:1788572535.350797:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000f940124a 429: 1< 0 - - > 2< 1757184 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.350800:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000000b3b738c 00000080:00200000:2.0:1788572535.350801:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 174. 00000080:00200000:2.0:1788572535.350802:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 430 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.350816:0:69530:0:(rw26.c:869:ll_write_end()) pos 1761280, len 4096, copied 4096 00000080:00200000:2.0:1788572535.350820:0:69530:0:(rw26.c:927:ll_write_end()) page@000000006b57a4a7[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 430] 00000080:00200000:2.0:1788572535.350823:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000e6e7163c 17ffffc0001001 3:0 ffff94d41257ba80 430 lru 00000080:00200000:2.0:1788572535.350828:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000b783c567 430: 1< 0 - - > 2< 1761280 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.350830:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000006b57a4a7 00000080:00200000:2.0:1788572535.350831:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 175. 00000080:00200000:2.0:1788572535.350833:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 431 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.350848:0:69530:0:(rw26.c:869:ll_write_end()) pos 1765376, len 4096, copied 4096 00000080:00200000:2.0:1788572535.350852:0:69530:0:(rw26.c:927:ll_write_end()) page@000000003217d77d[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 431] 00000080:00200000:2.0:1788572535.350854:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000031c6a120 17ffffc0001001 3:0 ffff94d41257b620 431 lru 00000080:00200000:2.0:1788572535.350859:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000018301cd2 431: 1< 0 - - > 2< 1765376 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.350862:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000003217d77d 00000080:00200000:2.0:1788572535.350863:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 176. 00000080:00200000:2.0:1788572535.350865:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 432 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.350878:0:69530:0:(rw26.c:869:ll_write_end()) pos 1769472, len 4096, copied 4096 00000080:00200000:2.0:1788572535.350882:0:69530:0:(rw26.c:927:ll_write_end()) page@000000009a83eb85[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 432] 00000080:00200000:2.0:1788572535.350885:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000be7cee19 17ffffc0001001 3:0 ffff94d41257b700 432 lru 00000080:00200000:2.0:1788572535.350889:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000070874462 432: 1< 0 - - > 2< 1769472 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.350891:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000009a83eb85 00000080:00200000:2.0:1788572535.350892:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 177. 00000080:00200000:2.0:1788572535.350893:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 433 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.350986:0:69530:0:(rw26.c:869:ll_write_end()) pos 1773568, len 4096, copied 4096 00000080:00200000:2.0:1788572535.350990:0:69530:0:(rw26.c:927:ll_write_end()) page@000000001308b86e[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 433] 00000080:00200000:2.0:1788572535.350993:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000ac933ba8 17ffffc0001001 3:0 ffff94d41257b9a0 433 lru 00000080:00200000:2.0:1788572535.350998:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000dc363594 433: 1< 0 - - > 2< 1773568 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.351001:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000001308b86e 00000080:00200000:2.0:1788572535.351002:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 178. 00000080:00200000:2.0:1788572535.351004:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 434 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.351017:0:69530:0:(rw26.c:869:ll_write_end()) pos 1777664, len 4096, copied 4096 00000080:00200000:2.0:1788572535.351020:0:69530:0:(rw26.c:927:ll_write_end()) page@000000006c2f330c[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 434] 00000080:00200000:2.0:1788572535.351023:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000c058ff5a 17ffffc0001001 3:0 ffff94d41257b1c0 434 lru 00000080:00200000:2.0:1788572535.351028:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000c700210a 434: 1< 0 - - > 2< 1777664 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.351030:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000006c2f330c 00000080:00200000:2.0:1788572535.351099:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 179. 00000080:00200000:2.0:1788572535.351104:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 435 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.351132:0:69530:0:(rw26.c:869:ll_write_end()) pos 1781760, len 4096, copied 4096 00000080:00200000:2.0:1788572535.351137:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000666c229b[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 435] 00000080:00200000:2.0:1788572535.351139:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000057bc08b2 17ffffc0001001 3:0 ffff94d4084ba380 435 lru 00000080:00200000:2.0:1788572535.351144:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000019c371df 435: 1< 0 - - > 2< 1781760 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.351146:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000666c229b 00000080:00200000:2.0:1788572535.351147:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 180. 00000080:00200000:2.0:1788572535.351149:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 436 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.351161:0:69530:0:(rw26.c:869:ll_write_end()) pos 1785856, len 4096, copied 4096 00000080:00200000:2.0:1788572535.351165:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000059bdeafa[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 436] 00000080:00200000:2.0:1788572535.351167:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000031414be0 17ffffc0001001 3:0 ffff94d4084ba460 436 lru 00000080:00200000:2.0:1788572535.351172:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000a5443ba5 436: 1< 0 - - > 2< 1785856 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.351174:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000059bdeafa 00000080:00200000:2.0:1788572535.351175:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 181. 00000080:00200000:2.0:1788572535.351176:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 437 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.351186:0:69530:0:(rw26.c:869:ll_write_end()) pos 1789952, len 4096, copied 4096 00000080:00200000:2.0:1788572535.351189:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000586b65f7[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 437] 00000080:00200000:2.0:1788572535.351192:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000606dd1c7 17ffffc0001001 3:0 ffff94d4084ba000 437 lru 00000080:00200000:2.0:1788572535.351196:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000855d672f 437: 1< 0 - - > 2< 1789952 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.351199:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000586b65f7 00000080:00200000:2.0:1788572535.351200:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 182. 00000080:00200000:2.0:1788572535.351427:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 438 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.351444:0:69530:0:(rw26.c:869:ll_write_end()) pos 1794048, len 4096, copied 4096 00000080:00200000:2.0:1788572535.351460:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000f1b699d2[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 438] 00000080:00200000:2.0:1788572535.351464:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000025b25c43 17ffffc0001001 3:0 ffff94d4084ba8c0 438 lru 00000080:00200000:2.0:1788572535.351469:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000ae56e2b7 438: 1< 0 - - > 2< 1794048 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.351470:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000f1b699d2 00000080:00200000:2.0:1788572535.351471:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 183. 00000080:00200000:2.0:1788572535.351473:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 439 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.351489:0:69530:0:(rw26.c:869:ll_write_end()) pos 1798144, len 4096, copied 4096 00000080:00200000:2.0:1788572535.351493:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000aa3c6e61[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 439] 00000080:00200000:2.0:1788572535.351496:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000c2a9fcc0 17ffffc0001001 3:0 ffff94d4084baee0 439 lru 00000080:00200000:2.0:1788572535.351501:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000025ab0e1c 439: 1< 0 - - > 2< 1798144 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.351503:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000aa3c6e61 00000080:00200000:2.0:1788572535.351504:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 184. 00000080:00200000:2.0:1788572535.351505:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 440 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.351519:0:69530:0:(rw26.c:869:ll_write_end()) pos 1802240, len 4096, copied 4096 00000080:00200000:2.0:1788572535.351523:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000081769123[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 440] 00000080:00200000:2.0:1788572535.351526:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000003ba41bd9 17ffffc0001001 3:0 ffff94d4084ba540 440 lru 00000080:00200000:2.0:1788572535.351531:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000040e1a49b 440: 1< 0 - - > 2< 1802240 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.351533:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000081769123 00000080:00200000:2.0:1788572535.351534:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 185. 00000080:00200000:2.0:1788572535.351536:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 441 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.351549:0:69530:0:(rw26.c:869:ll_write_end()) pos 1806336, len 4096, copied 4096 00000080:00200000:2.0:1788572535.351553:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000085dc4e50[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 441] 00000080:00200000:2.0:1788572535.351556:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000056d0899d 17ffffc0001001 3:0 ffff94d4084bae00 441 lru 00000080:00200000:2.0:1788572535.351561:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000043ae6238 441: 1< 0 - - > 2< 1806336 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.351562:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000085dc4e50 00000080:00200000:2.0:1788572535.351563:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 186. 00000080:00200000:2.0:1788572535.351565:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 442 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.351584:0:69530:0:(rw26.c:869:ll_write_end()) pos 1810432, len 4096, copied 4096 00000080:00200000:2.0:1788572535.351589:0:69530:0:(rw26.c:927:ll_write_end()) page@000000007a6e7241[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 442] 00000080:00200000:2.0:1788572535.351591:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000a3f245ab 17ffffc0001021 2:0 ffff94d4084ba0e0 442 lru 00000080:00200000:2.0:1788572535.351597:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000d6be4f32 442: 1< 0 - - > 2< 1810432 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.351598:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000007a6e7241 00000080:00200000:2.0:1788572535.351599:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 187. 00000080:00200000:2.0:1788572535.351601:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 443 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.351614:0:69530:0:(rw26.c:869:ll_write_end()) pos 1814528, len 4096, copied 4096 00000080:00200000:2.0:1788572535.351617:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000ce10d4e9[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 443] 00000080:00200000:2.0:1788572535.351620:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000068f0e958 17ffffc0001001 3:0 ffff94d4084ba2a0 443 lru 00000080:00200000:2.0:1788572535.351626:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000bc0d7503 443: 1< 0 - - > 2< 1814528 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.351628:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000ce10d4e9 00000080:00200000:2.0:1788572535.351629:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 188. 00000080:00200000:2.0:1788572535.351631:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 444 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.351646:0:69530:0:(rw26.c:869:ll_write_end()) pos 1818624, len 4096, copied 4096 00000080:00200000:2.0:1788572535.351650:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000f780e124[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 444] 00000080:00200000:2.0:1788572535.351652:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000008d3b2335 17ffffc0001001 3:0 ffff94d4084ba7e0 444 lru 00000080:00200000:2.0:1788572535.351657:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000d602bd45 444: 1< 0 - - > 2< 1818624 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.351659:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000f780e124 00000080:00200000:2.0:1788572535.351660:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 189. 00000080:00200000:2.0:1788572535.351662:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 445 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.351675:0:69530:0:(rw26.c:869:ll_write_end()) pos 1822720, len 4096, copied 4096 00000080:00200000:2.0:1788572535.351680:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000cec68764[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 445] 00000080:00200000:2.0:1788572535.351683:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000e7155cf9 17ffffc0001001 3:0 ffff94d4084bab60 445 lru 00000080:00200000:2.0:1788572535.351688:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000006c7034d 445: 1< 0 - - > 2< 1822720 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.351690:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000cec68764 00000080:00200000:2.0:1788572535.351691:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 190. 00000080:00200000:2.0:1788572535.351692:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 446 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.351706:0:69530:0:(rw26.c:869:ll_write_end()) pos 1826816, len 4096, copied 4096 00000080:00200000:2.0:1788572535.351710:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000306a7e90[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 446] 00000080:00200000:2.0:1788572535.351713:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000fcef40ce 17ffffc0001001 3:0 ffff94d4084baa80 446 lru 00000080:00200000:2.0:1788572535.351719:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000c52659d1 446: 1< 0 - - > 2< 1826816 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.351720:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000306a7e90 00000080:00200000:2.0:1788572535.351722:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 191. 00000080:00200000:2.0:1788572535.351724:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 447 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.351738:0:69530:0:(rw26.c:869:ll_write_end()) pos 1830912, len 4096, copied 4096 00000080:00200000:2.0:1788572535.351742:0:69530:0:(rw26.c:927:ll_write_end()) page@000000004953445f[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 447] 00000080:00200000:2.0:1788572535.351745:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000006e6a3b1f 17ffffc0001001 3:0 ffff94d4084ba620 447 lru 00000080:00200000:2.0:1788572535.351749:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000001ee31121 447: 1< 0 - - > 2< 1830912 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.351751:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000004953445f 00000080:00200000:2.0:1788572535.351752:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 192. 00000080:00200000:2.0:1788572535.351754:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 448 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.351775:0:69530:0:(rw26.c:869:ll_write_end()) pos 1835008, len 4096, copied 4096 00000080:00200000:2.0:1788572535.351779:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000044863e70[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 448] 00000080:00200000:2.0:1788572535.351782:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000743745c4 17ffffc0001001 3:0 ffff94d4084ba700 448 lru 00000080:00200000:2.0:1788572535.351786:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000028570aea 448: 1< 0 - - > 2< 1835008 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.351788:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000044863e70 00000080:00200000:2.0:1788572535.351789:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 193. 00000080:00200000:2.0:1788572535.351790:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 449 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.351804:0:69530:0:(rw26.c:869:ll_write_end()) pos 1839104, len 4096, copied 4096 00000080:00200000:2.0:1788572535.351808:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000176b03c8[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 449] 00000080:00200000:2.0:1788572535.351811:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000006a0771b2 17ffffc0001001 3:0 ffff94d4084ba9a0 449 lru 00000080:00200000:2.0:1788572535.351815:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000028c050fd 449: 1< 0 - - > 2< 1839104 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.351817:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000176b03c8 00000080:00200000:2.0:1788572535.351818:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 194. 00000080:00200000:2.0:1788572535.351820:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 450 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.351833:0:69530:0:(rw26.c:869:ll_write_end()) pos 1843200, len 4096, copied 4096 00000080:00200000:2.0:1788572535.351838:0:69530:0:(rw26.c:927:ll_write_end()) page@000000000c92f17e[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 450] 00000080:00200000:2.0:1788572535.351841:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000595448ee 17ffffc0001001 3:0 ffff94d4084ba1c0 450 lru 00000080:00200000:2.0:1788572535.351846:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000e2e978f4 450: 1< 0 - - > 2< 1843200 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.351848:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000000c92f17e 00000080:00200000:2.0:1788572535.351849:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 195. 00000080:00200000:2.0:1788572535.351851:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 451 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.351865:0:69530:0:(rw26.c:869:ll_write_end()) pos 1847296, len 4096, copied 4096 00000080:00200000:2.0:1788572535.351870:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000a6f4a8d5[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 451] 00000080:00200000:2.0:1788572535.351873:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000146bc200 17ffffc0001001 3:0 ffff94d4084bad20 451 lru 00000080:00200000:2.0:1788572535.351878:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000c777e77c 451: 1< 0 - - > 2< 1847296 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.351880:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000a6f4a8d5 00000080:00200000:2.0:1788572535.351881:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 196. 00000080:00200000:2.0:1788572535.351883:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 452 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.351969:0:69530:0:(rw26.c:869:ll_write_end()) pos 1851392, len 4096, copied 4096 00000080:00200000:2.0:1788572535.351975:0:69530:0:(rw26.c:927:ll_write_end()) page@000000007a9e0e0b[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 452] 00000080:00200000:2.0:1788572535.351978:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000c2781884 17ffffc0001001 3:0 ffff94d4084bac40 452 lru 00000080:00200000:2.0:1788572535.351985:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000004464802 452: 1< 0 - - > 2< 1851392 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.351987:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000007a9e0e0b 00000080:00200000:2.0:1788572535.351988:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 197. 00000080:00200000:2.0:1788572535.351990:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 453 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.352013:0:69530:0:(rw26.c:869:ll_write_end()) pos 1855488, len 4096, copied 4096 00000080:00200000:2.0:1788572535.352017:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000c49ab7af[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 453] 00000080:00200000:2.0:1788572535.352020:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000c81926c1 17ffffc0001001 3:0 ffff94d408f2fb60 453 lru 00000080:00200000:2.0:1788572535.352026:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000956085d6 453: 1< 0 - - > 2< 1855488 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.352028:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000c49ab7af 00000080:00200000:2.0:1788572535.352029:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 198. 00000080:00200000:2.0:1788572535.352095:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 454 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.352113:0:69530:0:(rw26.c:869:ll_write_end()) pos 1859584, len 4096, copied 4096 00000080:00200000:2.0:1788572535.352119:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000989027ac[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 454] 00000080:00200000:2.0:1788572535.352122:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000e0c8ff5c 17ffffc0001001 3:0 ffff94d408f2fa80 454 lru 00000080:00200000:2.0:1788572535.352127:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000068cf9eea 454: 1< 0 - - > 2< 1859584 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.352129:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000989027ac 00000080:00200000:2.0:1788572535.352130:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 199. 00000080:00200000:2.0:1788572535.352132:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 455 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.352145:0:69530:0:(rw26.c:869:ll_write_end()) pos 1863680, len 4096, copied 4096 00000080:00200000:2.0:1788572535.352149:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000e000a6bc[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 455] 00000080:00200000:2.0:1788572535.352152:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000ed9f734d 17ffffc0001001 3:0 ffff94d408f2f620 455 lru 00000080:00200000:2.0:1788572535.352157:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000532f560c 455: 1< 0 - - > 2< 1863680 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.352160:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000e000a6bc 00000080:00200000:2.0:1788572535.352162:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 200. 00000080:00200000:2.0:1788572535.352163:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 456 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.352179:0:69530:0:(rw26.c:869:ll_write_end()) pos 1867776, len 4096, copied 4096 00000080:00200000:2.0:1788572535.352184:0:69530:0:(rw26.c:927:ll_write_end()) page@000000000431d551[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 456] 00000080:00200000:2.0:1788572535.352187:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000bdc0ba0e 17ffffc0001001 3:0 ffff94d408f2f700 456 lru 00000080:00200000:2.0:1788572535.352192:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000089277d2a 456: 1< 0 - - > 2< 1867776 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.352195:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000000431d551 00000080:00200000:2.0:1788572535.352196:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 201. 00000080:00200000:2.0:1788572535.352198:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 457 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.352387:0:69530:0:(rw26.c:869:ll_write_end()) pos 1871872, len 4096, copied 4096 00000080:00200000:2.0:1788572535.352393:0:69530:0:(rw26.c:927:ll_write_end()) page@000000008ba2181a[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 457] 00000080:00200000:2.0:1788572535.352395:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000cc524637 17ffffc0001021 2:0 ffff94d408f2f9a0 457 lru 00000080:00200000:2.0:1788572535.352400:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000008208c754 457: 1< 0 - - > 2< 1871872 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.352402:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000008ba2181a 00000080:00200000:2.0:1788572535.352404:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 202. 00000080:00200000:2.0:1788572535.352406:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 458 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.352422:0:69530:0:(rw26.c:869:ll_write_end()) pos 1875968, len 4096, copied 4096 00000080:00200000:2.0:1788572535.352426:0:69530:0:(rw26.c:927:ll_write_end()) page@000000003fb666d8[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 458] 00000080:00200000:2.0:1788572535.352428:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000003b59c077 17ffffc0001001 3:0 ffff94d408f2f1c0 458 lru 00000080:00200000:2.0:1788572535.352433:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000005b358601 458: 1< 0 - - > 2< 1875968 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.352435:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000003fb666d8 00000080:00200000:2.0:1788572535.352436:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 203. 00000080:00200000:2.0:1788572535.352437:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 459 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.352447:0:69530:0:(rw26.c:869:ll_write_end()) pos 1880064, len 4096, copied 4096 00000080:00200000:2.0:1788572535.352461:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000e2883bd3[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 459] 00000080:00200000:2.0:1788572535.352464:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000f5afa895 17ffffc0001001 3:0 ffff94d408f2fd20 459 lru 00000080:00200000:2.0:1788572535.352469:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000003b3df422 459: 1< 0 - - > 2< 1880064 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.352471:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000e2883bd3 00000080:00200000:2.0:1788572535.352472:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 204. 00000080:00200000:2.0:1788572535.352474:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 460 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.352487:0:69530:0:(rw26.c:869:ll_write_end()) pos 1884160, len 4096, copied 4096 00000080:00200000:2.0:1788572535.352491:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000092e8c738[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 460] 00000080:00200000:2.0:1788572535.352494:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000588ee7d2 17ffffc0001001 3:0 ffff94d408f2fc40 460 lru 00000080:00200000:2.0:1788572535.352498:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000006d7a5f4f 460: 1< 0 - - > 2< 1884160 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.352500:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000092e8c738 00000080:00200000:2.0:1788572535.352502:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 205. 00000080:00200000:2.0:1788572535.352503:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 461 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.352513:0:69530:0:(rw26.c:869:ll_write_end()) pos 1888256, len 4096, copied 4096 00000080:00200000:2.0:1788572535.352516:0:69530:0:(rw26.c:927:ll_write_end()) page@000000001e4b14a3[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 461] 00000080:00200000:2.0:1788572535.352519:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000b07283fd 17ffffc0001001 3:0 ffff94d408f2f380 461 lru 00000080:00200000:2.0:1788572535.352524:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000000c6cf63e 461: 1< 0 - - > 2< 1888256 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.352525:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000001e4b14a3 00000080:00200000:2.0:1788572535.352527:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 206. 00000080:00200000:2.0:1788572535.352528:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 462 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.352539:0:69530:0:(rw26.c:869:ll_write_end()) pos 1892352, len 4096, copied 4096 00000080:00200000:2.0:1788572535.352543:0:69530:0:(rw26.c:927:ll_write_end()) page@000000002c41c65f[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 462] 00000080:00200000:2.0:1788572535.352546:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000b7e61db7 17ffffc0001001 3:0 ffff94d408f2f460 462 lru 00000080:00200000:2.0:1788572535.352550:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000a73325a2 462: 1< 0 - - > 2< 1892352 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.352553:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000002c41c65f 00000080:00200000:2.0:1788572535.352554:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 207. 00000080:00200000:2.0:1788572535.352556:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 463 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.352568:0:69530:0:(rw26.c:869:ll_write_end()) pos 1896448, len 4096, copied 4096 00000080:00200000:2.0:1788572535.352571:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000d5fe4478[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 463] 00000080:00200000:2.0:1788572535.352574:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000004019d3bc 17ffffc0001001 3:0 ffff94d408f2f000 463 lru 00000080:00200000:2.0:1788572535.352578:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000c20451ae 463: 1< 0 - - > 2< 1896448 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.352580:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000d5fe4478 00000080:00200000:2.0:1788572535.352581:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 208. 00000080:00200000:2.0:1788572535.352582:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 464 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.352595:0:69530:0:(rw26.c:869:ll_write_end()) pos 1900544, len 4096, copied 4096 00000080:00200000:2.0:1788572535.352599:0:69530:0:(rw26.c:927:ll_write_end()) page@000000009e2edee9[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 464] 00000080:00200000:2.0:1788572535.352602:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000167f9195 17ffffc0001001 3:0 ffff94d408f2f8c0 464 lru 00000080:00200000:2.0:1788572535.352607:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000000d27c2d7 464: 1< 0 - - > 2< 1900544 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.352609:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000009e2edee9 00000080:00200000:2.0:1788572535.352610:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 209. 00000080:00200000:2.0:1788572535.352612:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 465 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.352626:0:69530:0:(rw26.c:869:ll_write_end()) pos 1904640, len 4096, copied 4096 00000080:00200000:2.0:1788572535.352630:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000cac71e92[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 465] 00000080:00200000:2.0:1788572535.352633:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000005364a451 17ffffc0001001 3:0 ffff94d408f2fee0 465 lru 00000080:00200000:2.0:1788572535.352638:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000063276bf3 465: 1< 0 - - > 2< 1904640 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.352640:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000cac71e92 00000080:00200000:2.0:1788572535.352641:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 210. 00000080:00200000:2.0:1788572535.352643:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 466 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.352657:0:69530:0:(rw26.c:869:ll_write_end()) pos 1908736, len 4096, copied 4096 00000080:00200000:2.0:1788572535.352661:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000952dff3e[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 466] 00000080:00200000:2.0:1788572535.352663:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000df6fad90 17ffffc0001001 3:0 ffff94d408f2f540 466 lru 00000080:00200000:2.0:1788572535.352669:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000f0035035 466: 1< 0 - - > 2< 1908736 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.352670:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000952dff3e 00000080:00200000:2.0:1788572535.352671:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 211. 00000080:00200000:2.0:1788572535.352673:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 467 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.352687:0:69530:0:(rw26.c:869:ll_write_end()) pos 1912832, len 4096, copied 4096 00000080:00200000:2.0:1788572535.352692:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000047bf6e90[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 467] 00000080:00200000:2.0:1788572535.352695:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000de21966b 17ffffc0001001 3:0 ffff94d408f2fe00 467 lru 00000080:00200000:2.0:1788572535.352700:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000044dbd51d 467: 1< 0 - - > 2< 1912832 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.352702:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000047bf6e90 00000080:00200000:2.0:1788572535.352703:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 212. 00000080:00200000:2.0:1788572535.352705:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 468 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.352722:0:69530:0:(rw26.c:869:ll_write_end()) pos 1916928, len 4096, copied 4096 00000080:00200000:2.0:1788572535.352726:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000ed6ffff6[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 468] 00000080:00200000:2.0:1788572535.352729:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000cdec8bfa 17ffffc0001001 3:0 ffff94d408f2f0e0 468 lru 00000080:00200000:2.0:1788572535.352734:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000006ac1d7ce 468: 1< 0 - - > 2< 1916928 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.352736:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000ed6ffff6 00000080:00200000:2.0:1788572535.352738:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 213. 00000080:00200000:2.0:1788572535.352739:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 469 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.352755:0:69530:0:(rw26.c:869:ll_write_end()) pos 1921024, len 4096, copied 4096 00000080:00200000:2.0:1788572535.352759:0:69530:0:(rw26.c:927:ll_write_end()) page@000000005dac134f[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 469] 00000080:00200000:2.0:1788572535.352761:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000009901ac41 17ffffc0001001 3:0 ffff94d408f2f2a0 469 lru 00000080:00200000:2.0:1788572535.352767:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000000cdd3c37 469: 1< 0 - - > 2< 1921024 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.352769:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000005dac134f 00000080:00200000:2.0:1788572535.352770:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 214. 00000080:00200000:2.0:1788572535.352772:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 470 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.352788:0:69530:0:(rw26.c:869:ll_write_end()) pos 1925120, len 4096, copied 4096 00000080:00200000:2.0:1788572535.352793:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000024e537ba[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 470] 00000080:00200000:2.0:1788572535.352795:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000005dd1f721 17ffffc0001001 3:0 ffff94d408f2f7e0 470 lru 00000080:00200000:2.0:1788572535.352800:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000001b6cfb3b 470: 1< 0 - - > 2< 1925120 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.352802:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000024e537ba 00000080:00200000:2.0:1788572535.352803:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 215. 00000080:00200000:2.0:1788572535.352805:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 471 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.352826:0:69530:0:(rw26.c:869:ll_write_end()) pos 1929216, len 4096, copied 4096 00000080:00200000:2.0:1788572535.352831:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000175be385[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 471] 00000080:00200000:2.0:1788572535.352833:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000ba07e285 17ffffc0001001 3:0 ffff94d412518b60 471 lru 00000080:00200000:2.0:1788572535.352837:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000f981d35f 471: 1< 0 - - > 2< 1929216 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.352839:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000175be385 00000080:00200000:2.0:1788572535.352840:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 216. 00000080:00200000:2.0:1788572535.352842:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 472 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.352860:0:69530:0:(rw26.c:869:ll_write_end()) pos 1933312, len 4096, copied 4096 00000080:00200000:2.0:1788572535.352864:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000ce2cb514[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 472] 00000080:00200000:2.0:1788572535.352866:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000f40eb2b6 17ffffc0001021 2:0 ffff94d412518a80 472 lru 00000080:00200000:2.0:1788572535.352870:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000006f39545b 472: 1< 0 - - > 2< 1933312 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.352872:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000ce2cb514 00000080:00200000:2.0:1788572535.352873:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 217. 00000080:00200000:2.0:1788572535.352875:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 473 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.352889:0:69530:0:(rw26.c:869:ll_write_end()) pos 1937408, len 4096, copied 4096 00000080:00200000:2.0:1788572535.352893:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000609df228[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 473] 00000080:00200000:2.0:1788572535.353174:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000004d81b9dd 17ffffc0001001 3:0 ffff94d412518620 473 lru 00000080:00200000:2.0:1788572535.353180:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000500f75b0 473: 1< 0 - - > 2< 1937408 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.353182:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000609df228 00000080:00200000:2.0:1788572535.353184:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 218. 00000080:00200000:2.0:1788572535.353186:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 474 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.353413:0:69530:0:(rw26.c:869:ll_write_end()) pos 1941504, len 4096, copied 4096 00000080:00200000:2.0:1788572535.353421:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000795d3c77[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 474] 00000080:00200000:2.0:1788572535.353423:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000d3759dec 17ffffc0001001 3:0 ffff94d412518700 474 lru 00000080:00200000:2.0:1788572535.353429:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000013636fc6 474: 1< 0 - - > 2< 1941504 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.353431:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000795d3c77 00000080:00200000:2.0:1788572535.353431:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 219. 00000080:00200000:2.0:1788572535.353433:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 475 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.353459:0:69530:0:(rw26.c:869:ll_write_end()) pos 1945600, len 4096, copied 4096 00000080:00200000:2.0:1788572535.353464:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000bed99976[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 475] 00000080:00200000:2.0:1788572535.353467:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000006969e20d 17ffffc0001001 3:0 ffff94d4125189a0 475 lru 00000080:00200000:2.0:1788572535.353472:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000003de3d1bc 475: 1< 0 - - > 2< 1945600 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.353474:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000bed99976 00000080:00200000:2.0:1788572535.353475:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 220. 00000080:00200000:2.0:1788572535.353477:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 476 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.353491:0:69530:0:(rw26.c:869:ll_write_end()) pos 1949696, len 4096, copied 4096 00000080:00200000:2.0:1788572535.353495:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000dfbb88bb[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 476] 00000080:00200000:2.0:1788572535.353498:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000f5136221 17ffffc0001001 3:0 ffff94d4125181c0 476 lru 00000080:00200000:2.0:1788572535.353503:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000009bc44af2 476: 1< 0 - - > 2< 1949696 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.353505:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000dfbb88bb 00000080:00200000:2.0:1788572535.353506:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 221. 00000080:00200000:2.0:1788572535.353508:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 477 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.353523:0:69530:0:(rw26.c:869:ll_write_end()) pos 1953792, len 4096, copied 4096 00000080:00200000:2.0:1788572535.353527:0:69530:0:(rw26.c:927:ll_write_end()) page@000000006448fcf3[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 477] 00000080:00200000:2.0:1788572535.353530:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000001d26bf56 17ffffc0001001 3:0 ffff94d412518d20 477 lru 00000080:00200000:2.0:1788572535.353535:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000052984b2b 477: 1< 0 - - > 2< 1953792 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.353538:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000006448fcf3 00000080:00200000:2.0:1788572535.353539:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 222. 00000080:00200000:2.0:1788572535.353540:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 478 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.353555:0:69530:0:(rw26.c:869:ll_write_end()) pos 1957888, len 4096, copied 4096 00000080:00200000:2.0:1788572535.353559:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000030d09ed0[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 478] 00000080:00200000:2.0:1788572535.353562:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000b5870a1b 17ffffc0001001 3:0 ffff94d412518c40 478 lru 00000080:00200000:2.0:1788572535.353567:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000d01ef4ee 478: 1< 0 - - > 2< 1957888 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.353569:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000030d09ed0 00000080:00200000:2.0:1788572535.353570:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 223. 00000080:00200000:2.0:1788572535.353571:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 479 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.353584:0:69530:0:(rw26.c:869:ll_write_end()) pos 1961984, len 4096, copied 4096 00000080:00200000:2.0:1788572535.353588:0:69530:0:(rw26.c:927:ll_write_end()) page@000000008cce2325[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 479] 00000080:00200000:2.0:1788572535.353590:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000856a7068 17ffffc0001001 3:0 ffff94d412518380 479 lru 00000080:00200000:2.0:1788572535.353594:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000ee49c199 479: 1< 0 - - > 2< 1961984 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.353596:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000008cce2325 00000080:00200000:2.0:1788572535.353597:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 224. 00000080:00200000:2.0:1788572535.353599:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 480 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.353611:0:69530:0:(rw26.c:869:ll_write_end()) pos 1966080, len 4096, copied 4096 00000080:00200000:2.0:1788572535.353616:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000a6ca9196[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 480] 00000080:00200000:2.0:1788572535.353618:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000b9b51a91 17ffffc0001001 3:0 ffff94d412518460 480 lru 00000080:00200000:2.0:1788572535.353623:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000245681f5 480: 1< 0 - - > 2< 1966080 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.353625:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000a6ca9196 00000080:00200000:2.0:1788572535.353626:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 225. 00000080:00200000:2.0:1788572535.353628:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 481 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.353642:0:69530:0:(rw26.c:869:ll_write_end()) pos 1970176, len 4096, copied 4096 00000080:00200000:2.0:1788572535.353647:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000011a0a9ce[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 481] 00000080:00200000:2.0:1788572535.353650:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000b86e8bca 17ffffc0001001 3:0 ffff94d412518000 481 lru 00000080:00200000:2.0:1788572535.353655:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000c50ff3de 481: 1< 0 - - > 2< 1970176 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.353657:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000011a0a9ce 00000080:00200000:2.0:1788572535.353658:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 226. 00000080:00200000:2.0:1788572535.353660:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 482 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.353674:0:69530:0:(rw26.c:869:ll_write_end()) pos 1974272, len 4096, copied 4096 00000080:00200000:2.0:1788572535.353679:0:69530:0:(rw26.c:927:ll_write_end()) page@000000002e079734[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 482] 00000080:00200000:2.0:1788572535.353682:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000002d476e7a 17ffffc0001001 3:0 ffff94d4125188c0 482 lru 00000080:00200000:2.0:1788572535.353686:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000bdcc7381 482: 1< 0 - - > 2< 1974272 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.353688:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000002e079734 00000080:00200000:2.0:1788572535.353689:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 227. 00000080:00200000:2.0:1788572535.353691:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 483 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.353703:0:69530:0:(rw26.c:869:ll_write_end()) pos 1978368, len 4096, copied 4096 00000080:00200000:2.0:1788572535.353708:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000b98dd457[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 483] 00000080:00200000:2.0:1788572535.353711:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000071ef6e7d 17ffffc0001001 3:0 ffff94d412518ee0 483 lru 00000080:00200000:2.0:1788572535.353715:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000d4490f6b 483: 1< 0 - - > 2< 1978368 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.353718:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000b98dd457 00000080:00200000:2.0:1788572535.353719:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 228. 00000080:00200000:2.0:1788572535.353720:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 484 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.353737:0:69530:0:(rw26.c:869:ll_write_end()) pos 1982464, len 4096, copied 4096 00000080:00200000:2.0:1788572535.353741:0:69530:0:(rw26.c:927:ll_write_end()) page@000000000a1af092[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 484] 00000080:00200000:2.0:1788572535.353744:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000ed324808 17ffffc0001001 3:0 ffff94d412518540 484 lru 00000080:00200000:2.0:1788572535.353750:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000008a04a446 484: 1< 0 - - > 2< 1982464 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.353752:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000000a1af092 00000080:00200000:2.0:1788572535.353753:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 229. 00000080:00200000:2.0:1788572535.353755:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 485 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.353775:0:69530:0:(rw26.c:869:ll_write_end()) pos 1986560, len 4096, copied 4096 00000080:00200000:2.0:1788572535.353779:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000d5f20f7e[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 485] 00000080:00200000:2.0:1788572535.353781:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000ead6a6f6 17ffffc0001001 3:0 ffff94d412518e00 485 lru 00000080:00200000:2.0:1788572535.353786:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000058597e29 485: 1< 0 - - > 2< 1986560 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.353789:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000d5f20f7e 00000080:00200000:2.0:1788572535.353790:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 230. 00000080:00200000:2.0:1788572535.353791:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 486 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.353807:0:69530:0:(rw26.c:869:ll_write_end()) pos 1990656, len 4096, copied 4096 00000080:00200000:2.0:1788572535.353811:0:69530:0:(rw26.c:927:ll_write_end()) page@000000003f0f3a63[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 486] 00000080:00200000:2.0:1788572535.353814:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000051ee50b6 17ffffc0001001 3:0 ffff94d4125180e0 486 lru 00000080:00200000:2.0:1788572535.353820:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000980d67d5 486: 1< 0 - - > 2< 1990656 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.353822:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000003f0f3a63 00000080:00200000:2.0:1788572535.353823:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 231. 00000080:00200000:2.0:1788572535.353824:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 487 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.353843:0:69530:0:(rw26.c:869:ll_write_end()) pos 1994752, len 4096, copied 4096 00000080:00200000:2.0:1788572535.353848:0:69530:0:(rw26.c:927:ll_write_end()) page@000000004abde62f[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 487] 00000080:00200000:2.0:1788572535.353850:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000052f95075 17ffffc0001021 2:0 ffff94d4125182a0 487 lru 00000080:00200000:2.0:1788572535.353855:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000007b46c05a 487: 1< 0 - - > 2< 1994752 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.353857:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000004abde62f 00000080:00200000:2.0:1788572535.353858:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 232. 00000080:00200000:2.0:1788572535.353860:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 488 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.353876:0:69530:0:(rw26.c:869:ll_write_end()) pos 1998848, len 4096, copied 4096 00000080:00200000:2.0:1788572535.353880:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000dbb7af7e[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 488] 00000080:00200000:2.0:1788572535.353883:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000d9feaaa1 17ffffc0001001 3:0 ffff94d4125187e0 488 lru 00000080:00200000:2.0:1788572535.353887:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000001eb9d3a6 488: 1< 0 - - > 2< 1998848 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.353889:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000dbb7af7e 00000080:00200000:2.0:1788572535.353890:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 233. 00000080:00200000:2.0:1788572535.353891:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 489 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.353912:0:69530:0:(rw26.c:869:ll_write_end()) pos 2002944, len 4096, copied 4096 00000080:00200000:2.0:1788572535.353917:0:69530:0:(rw26.c:927:ll_write_end()) page@000000005253d4ba[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 489] 00000080:00200000:2.0:1788572535.353920:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000009864315f 17ffffc0001001 3:0 ffff94d40820a460 489 lru 00000080:00200000:2.0:1788572535.353925:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000005022fa91 489: 1< 0 - - > 2< 2002944 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.353927:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000005253d4ba 00000080:00200000:2.0:1788572535.353928:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 234. 00000080:00200000:2.0:1788572535.353930:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 490 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.353943:0:69530:0:(rw26.c:869:ll_write_end()) pos 2007040, len 4096, copied 4096 00000080:00200000:2.0:1788572535.353948:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000078758b57[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 490] 00000080:00200000:2.0:1788572535.353951:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000a5d8eb95 17ffffc0001001 3:0 ffff94d40820a000 490 lru 00000080:00200000:2.0:1788572535.353956:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000052615992 490: 1< 0 - - > 2< 2007040 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.353959:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000078758b57 00000080:00200000:2.0:1788572535.353960:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 235. 00000080:00200000:2.0:1788572535.353963:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 491 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.353979:0:69530:0:(rw26.c:869:ll_write_end()) pos 2011136, len 4096, copied 4096 00000080:00200000:2.0:1788572535.353983:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000031ed90b3[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 491] 00000080:00200000:2.0:1788572535.353986:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000dd6b5fe6 17ffffc0001001 3:0 ffff94d40820a8c0 491 lru 00000080:00200000:2.0:1788572535.353991:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000008824e5ee 491: 1< 0 - - > 2< 2011136 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.353993:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000031ed90b3 00000080:00200000:2.0:1788572535.353994:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 236. 00000080:00200000:2.0:1788572535.353996:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 492 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.354012:0:69530:0:(rw26.c:869:ll_write_end()) pos 2015232, len 4096, copied 4096 00000080:00200000:2.0:1788572535.354017:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000a68f1700[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 492] 00000080:00200000:2.0:1788572535.354019:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000000928b447 17ffffc0001001 3:0 ffff94d40820aee0 492 lru 00000080:00200000:2.0:1788572535.354024:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000acadc294 492: 1< 0 - - > 2< 2015232 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.354026:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000a68f1700 00000080:00200000:2.0:1788572535.354027:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 237. 00000080:00200000:2.0:1788572535.354029:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 493 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.354124:0:69530:0:(rw26.c:869:ll_write_end()) pos 2019328, len 4096, copied 4096 00000080:00200000:2.0:1788572535.354129:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000076ef5685[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 493] 00000080:00200000:2.0:1788572535.354132:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000076dae512 17ffffc0001001 3:0 ffff94d40820a540 493 lru 00000080:00200000:2.0:1788572535.354136:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000000373eea2 493: 1< 0 - - > 2< 2019328 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.354138:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000076ef5685 00000080:00200000:2.0:1788572535.354139:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 238. 00000080:00200000:2.0:1788572535.354141:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 494 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.354153:0:69530:0:(rw26.c:869:ll_write_end()) pos 2023424, len 4096, copied 4096 00000080:00200000:2.0:1788572535.354157:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000e08dc7d0[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 494] 00000080:00200000:2.0:1788572535.354161:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000009405c9ee 17ffffc0001001 3:0 ffff94d40820ae00 494 lru 00000080:00200000:2.0:1788572535.354165:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000001295feac 494: 1< 0 - - > 2< 2023424 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.354168:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000e08dc7d0 00000080:00200000:2.0:1788572535.354168:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 239. 00000080:00200000:2.0:1788572535.354170:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 495 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.354183:0:69530:0:(rw26.c:869:ll_write_end()) pos 2027520, len 4096, copied 4096 00000080:00200000:2.0:1788572535.354187:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000ffedda5a[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 495] 00000080:00200000:2.0:1788572535.354190:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000025257858 17ffffc0001001 3:0 ffff94d40820a0e0 495 lru 00000080:00200000:2.0:1788572535.354195:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000081be83cf 495: 1< 0 - - > 2< 2027520 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.354196:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000ffedda5a 00000080:00200000:2.0:1788572535.354197:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 240. 00000080:00200000:2.0:1788572535.354199:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 496 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.354382:0:69530:0:(rw26.c:869:ll_write_end()) pos 2031616, len 4096, copied 4096 00000080:00200000:2.0:1788572535.354388:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000768b9cc7[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 496] 00000080:00200000:2.0:1788572535.354391:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000a2f6e1c7 17ffffc0001001 3:0 ffff94d40820a2a0 496 lru 00000080:00200000:2.0:1788572535.354396:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000cf62b690 496: 1< 0 - - > 2< 2031616 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.354398:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000768b9cc7 00000080:00200000:2.0:1788572535.354399:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 241. 00000080:00200000:2.0:1788572535.354401:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 497 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.354417:0:69530:0:(rw26.c:869:ll_write_end()) pos 2035712, len 4096, copied 4096 00000080:00200000:2.0:1788572535.354421:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000ac63fcb3[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 497] 00000080:00200000:2.0:1788572535.354423:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000126fcf62 17ffffc0001001 3:0 ffff94d40820a7e0 497 lru 00000080:00200000:2.0:1788572535.354428:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000a3d4ffe4 497: 1< 0 - - > 2< 2035712 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.354430:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000ac63fcb3 00000080:00200000:2.0:1788572535.354431:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 242. 00000080:00200000:2.0:1788572535.354433:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 498 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.354459:0:69530:0:(rw26.c:869:ll_write_end()) pos 2039808, len 4096, copied 4096 00000080:00200000:2.0:1788572535.354464:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000059981c53[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 498] 00000080:00200000:2.0:1788572535.354466:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000000ac84f82 17ffffc0001001 3:0 ffff94d40820ab60 498 lru 00000080:00200000:2.0:1788572535.354472:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000047febb26 498: 1< 0 - - > 2< 2039808 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.354474:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000059981c53 00000080:00200000:2.0:1788572535.354475:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 243. 00000080:00200000:2.0:1788572535.354477:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 499 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.354494:0:69530:0:(rw26.c:869:ll_write_end()) pos 2043904, len 4096, copied 4096 00000080:00200000:2.0:1788572535.354499:0:69530:0:(rw26.c:927:ll_write_end()) page@0000000014864072[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 499] 00000080:00200000:2.0:1788572535.354501:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000009ff61744 17ffffc0001001 3:0 ffff94d40820aa80 499 lru 00000080:00200000:2.0:1788572535.354506:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000014bc147c 499: 1< 0 - - > 2< 2043904 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.354509:0:69530:0:(rw26.c:927:ll_write_end()) end page@0000000014864072 00000080:00200000:2.0:1788572535.354510:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 244. 00000080:00200000:2.0:1788572535.354511:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 500 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.354525:0:69530:0:(rw26.c:869:ll_write_end()) pos 2048000, len 4096, copied 4096 00000080:00200000:2.0:1788572535.354529:0:69530:0:(rw26.c:927:ll_write_end()) page@000000009e30de20[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 500] 00000080:00200000:2.0:1788572535.354531:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000a7e3a0f0 17ffffc0001001 3:0 ffff94d40820a620 500 lru 00000080:00200000:2.0:1788572535.354536:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@0000000042167e14 500: 1< 0 - - > 2< 2048000 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.354538:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000009e30de20 00000080:00200000:2.0:1788572535.354539:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 245. 00000080:00200000:2.0:1788572535.354541:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 501 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.354554:0:69530:0:(rw26.c:869:ll_write_end()) pos 2052096, len 4096, copied 4096 00000080:00200000:2.0:1788572535.354559:0:69530:0:(rw26.c:927:ll_write_end()) page@000000007ef0ea56[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 501] 00000080:00200000:2.0:1788572535.354562:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000099009089 17ffffc0001001 3:0 ffff94d40820a700 501 lru 00000080:00200000:2.0:1788572535.354567:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000009b10bec1 501: 1< 0 - - > 2< 2052096 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.354569:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000007ef0ea56 00000080:00200000:2.0:1788572535.354570:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 246. 00000080:00200000:2.0:1788572535.354572:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 502 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.354590:0:69530:0:(rw26.c:869:ll_write_end()) pos 2056192, len 4096, copied 4096 00000080:00200000:2.0:1788572535.354594:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000e71293c0[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 502] 00000080:00200000:2.0:1788572535.354597:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000041b282bd 17ffffc0001021 2:0 ffff94d40820a9a0 502 lru 00000080:00200000:2.0:1788572535.354602:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000610d2f01 502: 1< 0 - - > 2< 2056192 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.354604:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000e71293c0 00000080:00200000:2.0:1788572535.354605:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 247. 00000080:00200000:2.0:1788572535.354607:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 503 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.354624:0:69530:0:(rw26.c:869:ll_write_end()) pos 2060288, len 4096, copied 4096 00000080:00200000:2.0:1788572535.354628:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000b56b573f[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 503] 00000080:00200000:2.0:1788572535.354630:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000ed3408e3 17ffffc0001001 3:0 ffff94d40820a1c0 503 lru 00000080:00200000:2.0:1788572535.354635:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000a7db77dc 503: 1< 0 - - > 2< 2060288 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.354637:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000b56b573f 00000080:00200000:2.0:1788572535.354638:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 248. 00000080:00200000:2.0:1788572535.354640:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 504 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.354654:0:69530:0:(rw26.c:869:ll_write_end()) pos 2064384, len 4096, copied 4096 00000080:00200000:2.0:1788572535.354658:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000a207b716[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 504] 00000080:00200000:2.0:1788572535.354661:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000001f330940 17ffffc0001001 3:0 ffff94d40820ad20 504 lru 00000080:00200000:2.0:1788572535.354665:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000008907b21f 504: 1< 0 - - > 2< 2064384 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.354667:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000a207b716 00000080:00200000:2.0:1788572535.354668:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 249. 00000080:00200000:2.0:1788572535.354670:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 505 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.354685:0:69530:0:(rw26.c:869:ll_write_end()) pos 2068480, len 4096, copied 4096 00000080:00200000:2.0:1788572535.354689:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000f02c6209[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 505] 00000080:00200000:2.0:1788572535.354691:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000b6f359c2 17ffffc0001001 3:0 ffff94d40820ac40 505 lru 00000080:00200000:2.0:1788572535.354696:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000fe153ad0 505: 1< 0 - - > 2< 2068480 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.354698:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000f02c6209 00000080:00200000:2.0:1788572535.354699:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 250. 00000080:00200000:2.0:1788572535.354700:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 506 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.354712:0:69530:0:(rw26.c:869:ll_write_end()) pos 2072576, len 4096, copied 4096 00000080:00200000:2.0:1788572535.354716:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000b3d553e0[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 506] 00000080:00200000:2.0:1788572535.354718:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @0000000010ce56bb 17ffffc0001001 3:0 ffff94d40820a380 506 lru 00000080:00200000:2.0:1788572535.354722:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000a3da69f7 506: 1< 0 - - > 2< 2072576 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.354724:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000b3d553e0 00000080:00200000:2.0:1788572535.354725:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 251. 00000080:00200000:2.0:1788572535.354726:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 507 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.354744:0:69530:0:(rw26.c:869:ll_write_end()) pos 2076672, len 4096, copied 4096 00000080:00200000:2.0:1788572535.354748:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000ed2861a9[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 507] 00000080:00200000:2.0:1788572535.354751:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000006060057d 17ffffc0001001 3:0 ffff94d4121b20e0 507 lru 00000080:00200000:2.0:1788572535.354755:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000b9305ffe 507: 1< 0 - - > 2< 2076672 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.354757:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000ed2861a9 00000080:00200000:2.0:1788572535.354758:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 252. 00000080:00200000:2.0:1788572535.354760:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 508 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.354773:0:69530:0:(rw26.c:869:ll_write_end()) pos 2080768, len 4096, copied 4096 00000080:00200000:2.0:1788572535.354777:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000aac34bf8[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 508] 00000080:00200000:2.0:1788572535.354779:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @000000006a10e8ec 17ffffc0001001 3:0 ffff94d4121b22a0 508 lru 00000080:00200000:2.0:1788572535.354785:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@000000009c520b4c 508: 1< 0 - - > 2< 2080768 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.354787:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000aac34bf8 00000080:00200000:2.0:1788572535.354788:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 253. 00000080:00200000:2.0:1788572535.354790:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 509 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.354804:0:69530:0:(rw26.c:869:ll_write_end()) pos 2084864, len 4096, copied 4096 00000080:00200000:2.0:1788572535.354809:0:69530:0:(rw26.c:927:ll_write_end()) page@000000004b053471[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 509] 00000080:00200000:2.0:1788572535.354812:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000f13c3df4 17ffffc0001001 3:0 ffff94d4121b27e0 509 lru 00000080:00200000:2.0:1788572535.354817:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000e9bde42a 509: 1< 0 - - > 2< 2084864 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.354819:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000004b053471 00000080:00200000:2.0:1788572535.354820:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 254. 00000080:00200000:2.0:1788572535.354822:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 510 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.354837:0:69530:0:(rw26.c:869:ll_write_end()) pos 2088960, len 4096, copied 4096 00000080:00200000:2.0:1788572535.354841:0:69530:0:(rw26.c:927:ll_write_end()) page@000000008f69a100[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 510] 00000080:00200000:2.0:1788572535.354844:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000611cac62 17ffffc0001001 3:0 ffff94d4121b2b60 510 lru 00000080:00200000:2.0:1788572535.354849:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000a567d074 510: 1< 0 - - > 2< 2088960 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.354851:0:69530:0:(rw26.c:927:ll_write_end()) end page@000000008f69a100 00000080:00200000:2.0:1788572535.354852:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 255. 00000080:00200000:2.0:1788572535.354854:0:69530:0:(rw26.c:675:ll_write_begin()) Writing 511 of 0 to 4096 bytes 00000080:00200000:2.0:1788572535.354867:0:69530:0:(rw26.c:869:ll_write_end()) pos 2093056, len 4096, copied 4096 00000080:00200000:2.0:1788572535.354871:0:69530:0:(rw26.c:927:ll_write_end()) page@00000000cf7f67bf[3 0000000022bd7ff5 2 1 00000000f2c50006 65536 0 511] 00000080:00200000:2.0:1788572535.354874:0:69530:0:(rw26.c:927:ll_write_end()) vmpage @00000000b1b7e690 17ffffc0001001 3:0 ffff94d4121b2a80 511 lru 00000080:00200000:2.0:1788572535.354879:0:69530:0:(rw26.c:927:ll_write_end()) osc-page@00000000714e5969 511: 1< 0 - - > 2< 2093056 0 0 0x0 0x0 | 000000008dfbe5d4 00000000a0850ecf > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572535.354881:0:69530:0:(rw26.c:927:ll_write_end()) end page@00000000cf7f67bf 00000080:00200000:2.0:1788572535.354882:0:69530:0:(rw26.c:927:ll_write_end()) queued cl_page: 256. 00000080:00200000:2.0:1788572535.354886:0:69530:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 256, from 0, to 4096 prio 0 00000008:00010000:2.0:1788572535.354947:0:69530:0:(osc_cache.c:857:osc_extent_find()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.354973:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.354987:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.354996:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355004:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355012:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355019:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355027:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355117:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355124:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355137:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355145:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355154:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355163:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355171:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572535.355188:0:69530:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff94d410bc33f8, count 15, dirtied 15 00000008:00010000:2.0:1788572535.355196:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355447:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355467:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355481:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355489:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355497:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355506:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355513:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355521:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355528:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355535:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355547:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355554:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355561:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355568:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572535.355580:0:69530:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff94d410bc33f8, count 15, dirtied 15 00000008:00010000:2.0:1788572535.355585:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355593:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355600:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355608:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355621:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355629:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355637:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355644:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355652:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355660:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355668:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355675:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355689:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355697:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355706:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572535.355718:0:69530:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff94d410bc33f8, count 15, dirtied 15 00000008:00010000:2.0:1788572535.355724:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355732:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355741:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355749:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355758:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355772:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355781:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355789:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355797:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355805:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355813:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355821:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355831:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355844:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355851:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572535.355865:0:69530:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff94d410bc33f8, count 15, dirtied 15 00000008:00010000:2.0:1788572535.355871:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355880:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355889:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355897:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355905:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355912:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355925:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355932:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355940:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355947:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355954:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355962:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355968:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355975:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.355987:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572535.355998:0:69530:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff94d410bc33f8, count 15, dirtied 15 00000008:00010000:2.0:1788572535.356002:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356010:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356017:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356025:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356110:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356119:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356128:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356139:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356147:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356154:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356162:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356170:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356178:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356187:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356197:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572535.356387:0:69530:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff94d410bc33f8, count 15, dirtied 15 00000008:00010000:2.0:1788572535.356395:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356405:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356414:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356423:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356431:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356440:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356448:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356467:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356481:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356489:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356498:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356506:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356513:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356521:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356529:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572535.356542:0:69530:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff94d410bc33f8, count 15, dirtied 15 00000008:00010000:2.0:1788572535.356548:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356560:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356568:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356575:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356583:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356592:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356600:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356608:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356616:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356628:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356636:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356644:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356653:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356662:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356669:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572535.356682:0:69530:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff94d410bc33f8, count 15, dirtied 15 00000008:00010000:2.0:1788572535.356687:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356695:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356708:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356716:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356723:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356731:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356739:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356745:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356752:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356760:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356774:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356782:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356790:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356797:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356805:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572535.356819:0:69530:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff94d410bc33f8, count 15, dirtied 15 00000008:00010000:2.0:1788572535.356824:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356833:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356840:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356851:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356859:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356866:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356874:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356882:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356891:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356898:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356906:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356918:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356927:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356935:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356943:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572535.356957:0:69530:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff94d410bc33f8, count 15, dirtied 15 00000008:00010000:2.0:1788572535.356962:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356971:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356979:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.356986:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357000:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357008:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357017:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357027:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357118:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357128:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357137:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357146:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357161:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357170:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357179:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572535.357192:0:69530:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff94d410bc33f8, count 15, dirtied 15 00000008:00010000:2.0:1788572535.357198:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357458:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357469:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357477:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357486:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357498:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357506:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357516:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357524:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357531:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357540:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357548:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357556:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357568:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357579:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572535.357593:0:69530:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff94d410bc33f8, count 15, dirtied 15 00000008:00010000:2.0:1788572535.357599:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357608:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357616:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357624:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357633:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357641:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357656:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357665:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357672:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357681:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357689:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357698:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357705:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357714:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357727:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572535.357740:0:69530:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff94d410bc33f8, count 15, dirtied 15 00000008:00010000:2.0:1788572535.357745:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357753:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357762:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357771:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357779:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357787:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357794:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357807:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357816:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357825:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357833:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357840:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357847:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357855:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357863:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572535.357880:0:69530:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff94d410bc33f8, count 15, dirtied 15 00000008:00010000:2.0:1788572535.357885:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357894:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357903:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357910:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357919:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357929:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357937:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357946:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357959:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357968:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357977:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357985:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.357993:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.358002:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.358010:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572535.358023:0:69530:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff94d410bc33f8, count 15, dirtied 15 00000008:00010000:2.0:1788572535.358028:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.358121:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.358131:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.358139:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.358147:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.358155:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.358164:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.358171:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.358179:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.358193:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.358295:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.358375:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.358389:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.358397:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.358405:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572535.358419:0:69530:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff94d410bc33f8, count 15, dirtied 15 00000008:00010000:2.0:1788572535.358425:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.358435:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.358447:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.358466:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.358475:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.358484:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.358491:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.358499:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.358508:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.358516:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.358529:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.358536:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.358545:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.358554:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572535.358562:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572535.358576:0:69530:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff94d410bc33f8, count 15, dirtied 15 00000008:00010000:2.0:1788572535.358582:0:69530:0:(osc_cache.c:2569:osc_queue_async_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572535.358592:0:69530:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff94d410bc33f8, count 1, dirtied 1 00000080:00200000:2.0:1788572535.358596:0:69530:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 256 pages 1048576 bytes, tot: 1048576 00000080:00200000:2.0:1788572535.358601:0:69530:0:(vvp_io.c:1358:vvp_io_write_start()) f132a.replay-single: write bytes 0, result: 1048576 00000020:00010000:2.0:1788572535.358617:0:69530:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff94d40392ed58 (ffff94d405cee488/0) at cl_lock_release():245 00000020:00010000:2.0:1788572535.358620:0:69530:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d40392ed58 (ffff94d405cee488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572535.358624:0:69530:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d4057bd730 (ffff94d405cee1b8/1) at cl_lock_cancel():142 00010000:00010000:2.0:1788572535.358633:0:69530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 5/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572535.358643:0:69530:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,0 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00010000:2.0:1788572535.358665:0:69530:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d40392ed58 (ffff94d405cee488/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572535.358668:0:69530:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d4057bd730 (ffff94d405cee488/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572535.358678:0:69530:0:(vvp_io.c:533:vvp_io_advance()) advancing 1048576 bytes 00000080:00200000:2.0:1788572535.358685:0:69530:0:(file.c:2205:ll_file_io_generic()) Range unlock [256, 511] 00000080:00200000:2.0:1788572535.358718:0:69530:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x9:0x0] ignore/verify layout 0/0, layout version 3 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572535.358722:0:69530:0:(file.c:2238:ll_file_io_generic()) f132a.replay-single: 2 io complete with rc: 0, result: 1048576, restart: 0 00000080:00200000:2.0:1788572535.358734:0:69530:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000a811:0x9:0x0] updating i_size 2097152 i_blocks 0 00000080:00200000:2.0:1788572535.358738:0:69530:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 1048576 00000080:00200008:2.0:1788572535.358789:0:69530:0:(file.c:479:ll_file_release()) START file f132a.replay-single:[0x20000a811:0x9:0x0](ffff94d410bc3248), flags 100002 00000002:00100000:2.0:1788572535.358857:0:69530:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff94d408d3e680 x1875450488521344/t107374182410(107374182410) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 776/848 e 0 to 0 dl 1788572551 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572535.358870:0:69530:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572535.358876:0:69530:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c63b80 x1875450488523008/t0(0) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572535.358899:0:69530:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job dd:9bf287c0-f582-4b83-84c1-15e766218620:69530:1875450488523008:192.168.202.125@tcp:35:dd.0 00000100:00100000:2.0:1788572535.359107:0:69530:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d407e133c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572535.364169:0:69530:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job dd:9bf287c0-f582-4b83-84c1-15e766218620:69530:1875450488523008:192.168.202.125@tcp:35:dd.0 00000002:00100000:1.0:1788572535.364525:0:69530:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff94d408d3e680 x1875450488521344/t107374182410(107374182410) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 776/848 e 0 to 0 dl 1788572551 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572535.387823:0:69531:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:0.0:1788572535.387859:0:69531:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572535.387924:0:69531:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167400/0x1e260726fbbc0b46 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 69531 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.387943:0:69531:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167400/0x1e260726fbbc0b46 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 69531 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.387954:0:69531:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167400/0x1e260726fbbc0b46 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 69531 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572535.387979:0:69531:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69531:1875450488523136:192.168.202.125@tcp:101:lfs.0 00000100:00100000:0.0:1788572535.388553:0:69531:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416e1ff00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572535.401879:0:69531:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69531:1875450488523136:192.168.202.125@tcp:101:lfs.0 00010000:00010000:0.0:1788572535.401911:0:69531:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167400/0x1e260726fbbc0b46 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0a16 expref: -99 pid: 69531 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.401925:0:69531:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167400/0x1e260726fbbc0b46 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0a16 expref: -99 pid: 69531 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.401937:0:69531:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167400/0x1e260726fbbc0b46 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0a16 expref: -99 pid: 69531 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.401948:0:69531:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167400/0x1e260726fbbc0b46 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0a16 expref: -99 pid: 69531 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.401959:0:69531:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167400/0x1e260726fbbc0b46 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0a16 expref: -99 pid: 69531 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.401968:0:69531:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167400/0x1e260726fbbc0b46 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0a16 expref: -99 pid: 69531 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.401982:0:69531:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167400/0x1e260726fbbc0b46 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0a16 expref: -99 pid: 69531 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572535.401993:0:69531:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408d3c000 x1875450488523136/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572551 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572535.402007:0:69531:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167400/0x1e260726fbbc0b46 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0a16 expref: -99 pid: 69531 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.402019:0:69531:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788572535.402026:0:69531:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788572535.402066:0:69531:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1788572535.402071:0:69531:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0b46 00000080:00010000:0.0:1788572535.402076:0:69531:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572535.402079:0:69531:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b46 from it ffffb68a42047bf8 00010000:00010000:0.0:1788572535.402082:0:69531:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167400/0x1e260726fbbc0b46 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0a16 expref: -99 pid: 69531 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.402092:0:69531:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167400/0x1e260726fbbc0b46 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0a16 expref: -99 pid: 69531 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572535.402114:0:69531:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572535.402135:0:69531:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f132a.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), file ffff94d403b55b00, open_flags 18800, mode 0 opened 0 00000080:00200000:0.0:1788572535.402142:0:69531:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f132a.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), intent=open 00000002:00010000:0.0:1788572535.402164:0:69531:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f132a.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0304001 00010000:00010000:0.0:1788572535.402237:0:69531:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166000/0x1e260726fbbc0b4d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 69531 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572535.402246:0:69531:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166000/0x1e260726fbbc0b4d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 69531 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572535.402257:0:69531:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572535.402263:0:69531:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408d3c000 x1875450488523392/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 800/131992 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572535.402272:0:69531:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166000/0x1e260726fbbc0b4d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 69531 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572535.402298:0:69531:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69531:1875450488523392:192.168.202.125@tcp:101:lfs.0 00000100:00100000:1.0:1788572535.402861:0:69531:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416e1ff00 going to sleep for 11 seconds 00000100:00100000:1.0:1788572535.415940:0:69531:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69531:1875450488523392:192.168.202.125@tcp:101:lfs.0 00010000:00010000:1.0:1788572535.415969:0:69531:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166000/0x1e260726fbbc0b4d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 69531 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572535.415984:0:69531:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166000/0x1e260726fbbc0b4d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 69531 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572535.415991:0:69531:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166000/0x1e260726fbbc0b4d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 69531 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572535.415998:0:69531:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166000/0x1e260726fbbc0b4d lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 69531 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572535.416006:0:69531:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166000/0x1e260726fbbc0b4d lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 69531 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572535.416012:0:69531:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412166000 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:1.0:1788572535.416016:0:69531:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166000/0x1e260726fbbc0b4d lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 69531 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572535.416023:0:69531:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166000/0x1e260726fbbc0b4d lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 69531 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572535.416041:0:69531:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166000/0x1e260726fbbc0b4d lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 69531 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572535.416049:0:69531:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572535.416058:0:69531:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166000/0x1e260726fbbc0b4d lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 69531 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572535.416064:0:69531:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166000/0x1e260726fbbc0b4d lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 69531 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572535.416084:0:69531:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0xb, status=-20 req@ffff94d408d3c000 x1875450488523392/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 800/608 e 0 to 0 dl 1788572551 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572535.416098:0:69531:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc -20 00000080:00200000:1.0:1788572535.416277:0:69531:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572535.416298:0:69531:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572535.416312:0:69531:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167400/0x1e260726fbbc0b46 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0a16 expref: -99 pid: 69531 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572535.416325:0:69531:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167400/0x1e260726fbbc0b46 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0a16 expref: -99 pid: 69531 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572535.416333:0:69531:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572535.416338:0:69531:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572535.416342:0:69531:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0b46 00000080:00010000:1.0:1788572535.416347:0:69531:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b46 from it ffffb68a42047bb0 00010000:00010000:1.0:1788572535.416350:0:69531:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167400/0x1e260726fbbc0b46 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0a16 expref: -99 pid: 69531 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572535.416359:0:69531:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167400/0x1e260726fbbc0b46 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0a16 expref: -99 pid: 69531 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572535.416375:0:69531:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 44 00000080:00200008:1.0:1788572535.416386:0:69531:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 304000 00000080:00200080:1.0:1788572535.416422:0:69531:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c0086915 arg=7ba3d0 00000100:00100000:1.0:1788572535.416495:0:69531:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69531:1875450488523520:192.168.202.125@tcp:34:lfs.0 00000100:00100000:1.0:1788572535.416571:0:69531:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408c8b6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572535.420422:0:69531:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69531:1875450488523520:192.168.202.125@tcp:34:lfs.0 00000080:00200008:1.0:1788572535.420672:0:69531:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 304000 00010000:00010000:0.0:1788572535.429492:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1788572535.457737:0:69533:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:3.0:1788572535.457768:0:69533:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572535.457786:0:69533:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167400/0x1e260726fbbc0b46 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0a16 expref: -99 pid: 69531 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572535.457801:0:69533:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167400/0x1e260726fbbc0b46 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0a16 expref: -99 pid: 69531 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572535.457808:0:69533:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572535.457816:0:69533:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:3.0:1788572535.457820:0:69533:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0b46 00000080:00010000:3.0:1788572535.457828:0:69533:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b46 from it ffffb68a4208fbf8 00010000:00010000:3.0:1788572535.457832:0:69533:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167400/0x1e260726fbbc0b46 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0a16 expref: -99 pid: 69531 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572535.457841:0:69533:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167400/0x1e260726fbbc0b46 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0a16 expref: -99 pid: 69531 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788572535.457856:0:69533:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:3.0:1788572535.457867:0:69533:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f132a.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), file ffff94d40864b080, open_flags 8000, mode 0 opened 0 00000080:00200000:3.0:1788572535.457873:0:69533:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f132a.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), intent=open 00000002:00010000:3.0:1788572535.457888:0:69533:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f132a.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100001 00010000:00010000:3.0:1788572535.460179:0:69533:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031600/0x1e260726fbbc0b54 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 69533 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572535.460194:0:69533:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031600/0x1e260726fbbc0b54 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 69533 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788572535.460599:0:69533:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788572535.460612:0:69533:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c55180 x1875450488523776/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 800/131992 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788572535.460623:0:69533:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031600/0x1e260726fbbc0b54 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 69533 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572535.460648:0:69533:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job md5sum:9bf287c0-f582-4b83-84c1-15e766218620:69533:1875450488523776:192.168.202.125@tcp:101:md5sum.0 00000100:00100000:3.0:1788572535.461493:0:69533:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405411f00 going to sleep for 11 seconds 00000100:00100000:1.0:1788572535.464799:0:69533:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job md5sum:9bf287c0-f582-4b83-84c1-15e766218620:69533:1875450488523776:192.168.202.125@tcp:101:md5sum.0 00010000:00010000:1.0:1788572535.464830:0:69533:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031600/0x1e260726fbbc0b54 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0a5c expref: -99 pid: 69533 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572535.464849:0:69533:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031600/0x1e260726fbbc0b54 lrc: 4/1,0 mode: --/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0a5c expref: -99 pid: 69533 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572535.464857:0:69533:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031600/0x1e260726fbbc0b54 lrc: 4/1,0 mode: --/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0a5c expref: -99 pid: 69533 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572535.464870:0:69533:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031600/0x1e260726fbbc0b54 lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0a5c expref: -99 pid: 69533 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572535.464883:0:69533:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031600/0x1e260726fbbc0b54 lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0a5c expref: -99 pid: 69533 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572535.464912:0:69533:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031600/0x1e260726fbbc0b54 lrc: 5/2,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0a5c expref: -99 pid: 69533 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572535.464922:0:69533:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031600/0x1e260726fbbc0b54 lrc: 5/2,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0a5c expref: -99 pid: 69533 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572535.464936:0:69533:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031600/0x1e260726fbbc0b54 lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0a5c expref: -99 pid: 69533 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572535.464950:0:69533:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff94d408c55180 x1875450488523776/t107374182415(107374182415) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 800/872 e 0 to 0 dl 1788572551 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'md5sum.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572535.464963:0:69533:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d408c55180 x1875450488523776/t107374182415(107374182415) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 800/872 e 0 to 0 dl 1788572551 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'md5sum.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572535.464978:0:69533:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 264 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031600/0x1e260726fbbc0b54 lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0a5c expref: -99 pid: 69533 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788572535.464992:0:69533:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031600/0x1e260726fbbc0b54 lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0a5c expref: -99 pid: 69533 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572535.465003:0:69533:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589977617/9 (0 0) 00000080:00200000:1.0:1788572535.465010:0:69533:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572535.465108:0:69533:0:(namei.c:147:ll_iget()) got inode: [0x20000a811:0x9:0x0](ffff94d410bc3248): rc = 0 00000080:00200000:1.0:1788572535.465132:0:69533:0:(file.c:6826:ll_layout_conf()) [0x20000a811:0x9:0x0]: layout version change: 3 -> 3 00000080:00010000:1.0:1788572535.465143:0:69533:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a811:0x9:0x0](ffff94d410bc3248) for lock 0x1e260726fbbc0b54 00000080:00010000:1.0:1788572535.465153:0:69533:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a811:0x9:0x0](ffff94d410bc3248) 00000080:00200008:1.0:1788572535.465166:0:69533:0:(file.c:1090:ll_file_open()) START file f132a.replay-single:[0x20000a811:0x9:0x0](ffff94d410bc3248), flags 100000 00000002:00100000:1.0:1788572535.465188:0:69533:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d408c55180 x1875450488523776/t107374182415(107374182415) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 800/872 e 0 to 0 dl 1788572551 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'md5sum.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788572535.465278:0:69533:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b54 from it ffff94d4049b1060 00010000:00010000:1.0:1788572535.465283:0:69533:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031600/0x1e260726fbbc0b54 lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4e0a5c expref: -99 pid: 69533 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572535.465308:0:69533:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031600/0x1e260726fbbc0b54 lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e0a5c expref: -99 pid: 69533 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572535.465336:0:69533:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x20000a811:0x9:0x0](ffff94d410bc3248), inode mode 81a4 mask 44 00000080:00200008:1.0:1788572535.465407:0:69533:0:(file.c:6285:ll_getattr_dentry()) START file f132a.replay-single:[0x20000a811:0x9:0x0](ffff94d410bc3248), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572535.465414:0:69533:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x20000a811:0x9:0x0](ffff94d410bc3248),name=f132a.replay-single 00000002:00010000:1.0:1788572535.465435:0:69533:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x20000a811:0x9:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572535.465445:0:69533:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031600/0x1e260726fbbc0b54 lrc: 2/1,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e0a5c expref: -99 pid: 69533 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572535.465467:0:69533:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031600/0x1e260726fbbc0b54 lrc: 2/1,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e0a5c expref: -99 pid: 69533 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572535.465474:0:69533:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572535.465479:0:69533:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a811:0x9:0x0](ffff94d410bc3248) 00000080:00010000:1.0:1788572535.465482:0:69533:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a811:0x9:0x0](ffff94d410bc3248) for lock 0x1e260726fbbc0b54 00000080:00010000:1.0:1788572535.465486:0:69533:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b54 from it ffffb68a4208fcd8 00010000:00010000:1.0:1788572535.465490:0:69533:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031600/0x1e260726fbbc0b54 lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4e0a5c expref: -99 pid: 69533 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572535.465498:0:69533:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031600/0x1e260726fbbc0b54 lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e0a5c expref: -99 pid: 69533 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572535.465527:0:69533:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x9:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:1.0:1788572535.465539:0:69533:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000a811:0x9:0x0] 00000008:00010000:1.0:1788572535.465568:0:69533:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1788572535.465572:0:69533:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff94d416fa6830, osc lock ffff94d4124bddb0, flags 0x400001002 00000008:00010000:1.0:1788572535.465588:0:69533:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1788572535.465590:0:69533:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff94d416fa6870, osc lock ffff94d4124bc720, flags 0x400001002 00000008:00010000:1.0:1788572535.465603:0:69533:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1788572535.465604:0:69533:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff94d416fa68b0, osc lock ffff94d4124bc260, flags 0x400001002 00010000:00010000:1.0:1788572535.465622:0:69533:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572535.465635:0:69533:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 1048575) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:1.0:1788572535.465646:0:69533:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:1.0:1788572535.465650:0:69533:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572535.465658:0:69533:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572535.465678:0:69533:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 3/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788572535.465686:0:69533:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (1048576 18446744073709551615) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 3/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00200000:1.0:1788572535.465694:0:69533:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:1.0:1788572535.465697:0:69533:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 6/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788572535.465709:0:69533:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 5/0,0 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572535.465723:0:69533:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x703:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a08 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572535.465732:0:69533:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x703:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a08 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:1.0:1788572535.465740:0:69533:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:1.0:1788572535.465743:0:69533:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x703:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a08 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572535.465750:0:69533:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x703:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a08 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:1.0:1788572535.465758:0:69533:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1788572535.465767:0:69533:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000a811:0x9:0x0] updating i_size 2097152 i_blocks 0 00000020:00010000:1.0:1788572535.465774:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff94d412aa0b90 (ffff94d414ddaea8/0) at cl_lock_release():245 00000020:00010000:1.0:1788572535.465777:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d412aa0b90 (ffff94d414ddaea8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788572535.465780:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d416fa6830 (ffff94d414dda1b8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788572535.465784:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d416fa6870 (ffff94d414ddaa28/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788572535.465786:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d416fa68b0 (ffff94d414ddabd8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788572535.465789:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d412aa0b90 (ffff94d414ddaea8/0) at cl_lock_fini():72 00000020:00010000:1.0:1788572535.465792:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d416fa6830 (ffff94d414ddaea8/1) at cl_lock_fini():72 00000020:00010000:1.0:1788572535.465797:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d416fa6870 (ffff94d414ddaea8/1) at cl_lock_fini():72 00000020:00010000:1.0:1788572535.465800:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d416fa68b0 (ffff94d414ddaea8/1) at cl_lock_fini():72 00000080:00200000:1.0:1788572535.465835:0:69533:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x9:0x0] ignore/verify layout 0/1, layout version 3 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1788572535.465868:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 0, count: 32768 00000080:00200000:1.0:1788572535.465893:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 0 00000080:00200000:1.0:1788572535.465918:0:69533:0:(file.c:2090:ll_file_io_generic()) f132a.replay-single: read ppos: 0, bytes: 32768 00000080:00200000:1.0:1788572535.465923:0:69533:0:(file.c:1848:ll_io_set_mirror()) f132a.replay-single: desiginated mirror: 0 00000020:00200000:1.0:1788572535.465929:0:69533:0:(cl_io.c:214:cl_io_rw_init()) header@0000000099bd4c4d[0x110, 257, [0x20000a811:0x9:0x0] hash] 00000020:00200000:1.0:1788572535.465931:0:69533:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 32768) 0 0 00000080:00200000:1.0:1788572535.465934:0:69533:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572535.465946:0:69533:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 32768) 32768, 32768 00020000:00200000:1.0:1788572535.465951:0:69533:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572535.465971:0:69533:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 7] 00000008:00010000:1.0:1788572535.465978:0:69533:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:1.0:1788572535.465982:0:69533:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff94d408c7ea30, osc lock ffff94d4124bc260, flags 0x0 00010000:00010000:1.0:1788572535.465991:0:69533:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572535.466000:0:69533:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 32767) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572535.466009:0:69533:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 6/1,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572535.466017:0:69533:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 6/1,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572535.466024:0:69533:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 5/1,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572535.466032:0:69533:0:(vvp_io.c:835:vvp_io_read_start()) f132a.replay-single: read [0, 32768) 00000080:00200000:1.0:1788572535.466043:0:69533:0:(vvp_io.c:194:vvp_prep_size()) [0x20000a811:0x9:0x0] updating i_size 2097152 00000080:00200000:1.0:1788572535.466047:0:69533:0:(vvp_io.c:893:vvp_io_read_start()) count:32768 iocb pos:0 00010000:00010000:1.0:1788572535.466079:0:69533:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 4/1,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572535.466088:0:69533:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (4096 8191) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 4/1,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572535.466097:0:69533:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 7/2,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572535.466105:0:69533:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 7/2,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572535.466112:0:69533:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 6/2,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:1.0:1788572535.466261:0:69533:0:(osc_cache.c:3010:osc_queue_sync_pages()) ### no dlm lock: extent: ffff94d416fa1840 00010000:00010000:1.0:1788572535.466281:0:69533:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 6/2,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572535.466289:0:69533:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 5/1,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:1.0:1788572535.466336:0:68272:0:(osc_cache.c:2022:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff94d416fa1840 00010000:00010000:1.0:1788572535.466357:0:68272:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 4/1,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00100000:1.0:1788572535.466400:0:68272:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff94d4049bc380 - object 0x280000401:1794 offset 0<>32768 00000100:00100000:1.0:1788572535.466445:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788572535.466476:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488523904:192.168.202.125@tcp:3:md5sum.0 00000008:00010000:0.0:1788572535.479192:0:2403:0:(osc_cache.c:910:osc_extent_finish()) ### no dlm lock: extent: ffff94d416fa1840 00000020:00200000:0.0:1788572535.481331:0:2403:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1788572535.481410:0:2403:0:(osc_cache.c:496:__osc_extent_remove()) ### no dlm lock: extent: ffff94d416fa1840 00000100:00100000:0.0:1788572535.481446:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488523904:192.168.202.125@tcp:3:md5sum.0 00000020:00010000:1.0:1788572535.487411:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff94d412304038 (ffff94d414ddaea8/0) at cl_lock_release():245 00000020:00010000:1.0:1788572535.487422:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d412304038 (ffff94d414ddaea8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788572535.487426:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d408c7ea30 (ffff94d414ddabd8/1) at cl_lock_cancel():142 00010000:00010000:1.0:1788572535.487437:0:69533:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 4/1,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572535.487463:0:69533:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000020:00010000:1.0:1788572535.487481:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d412304038 (ffff94d414ddaea8/0) at cl_lock_fini():72 00000020:00010000:1.0:1788572535.487485:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d408c7ea30 (ffff94d414ddaea8/1) at cl_lock_fini():72 00000080:00200000:1.0:1788572535.487494:0:69533:0:(vvp_io.c:533:vvp_io_advance()) advancing 32768 bytes 00000080:00200000:1.0:1788572535.487529:0:69533:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x9:0x0] ignore/verify layout 0/0, layout version 3 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572535.487534:0:69533:0:(file.c:2238:ll_file_io_generic()) f132a.replay-single: 1 io complete with rc: 0, result: 32768, restart: 0 00000080:00200000:1.0:1788572535.487538:0:69533:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 32768 00000080:00200008:1.0:1788572535.487633:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 32768, count: 32768 00000080:00200000:1.0:1788572535.487655:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 8 00000080:00200000:1.0:1788572535.487689:0:69533:0:(file.c:2090:ll_file_io_generic()) f132a.replay-single: read ppos: 32768, bytes: 32768 00000080:00200000:1.0:1788572535.487696:0:69533:0:(file.c:1848:ll_io_set_mirror()) f132a.replay-single: desiginated mirror: 0 00000020:00200000:1.0:1788572535.487702:0:69533:0:(cl_io.c:214:cl_io_rw_init()) header@0000000099bd4c4d[0x110, 265, [0x20000a811:0x9:0x0] hash] 00000020:00200000:1.0:1788572535.487704:0:69533:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [32768, 65536) 0 0 00000080:00200000:1.0:1788572535.487707:0:69533:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572535.487722:0:69533:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 32768 chunk: [32768, 65536) 65536, 32768 00020000:00200000:1.0:1788572535.487727:0:69533:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572535.487752:0:69533:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [8, 15] 00000008:00010000:1.0:1788572535.487763:0:69533:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:1.0:1788572535.487768:0:69533:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff94d408c7ea30, osc lock ffff94d4124bc260, flags 0x0 00010000:00010000:1.0:1788572535.487789:0:69533:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572535.487803:0:69533:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (32768 65535) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572535.487815:0:69533:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 6/1,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572535.487822:0:69533:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 6/1,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572535.487830:0:69533:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 5/1,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000080:00200000:1.0:1788572535.487839:0:69533:0:(vvp_io.c:835:vvp_io_read_start()) f132a.replay-single: read [32768, 65536) 00000080:00200000:1.0:1788572535.487850:0:69533:0:(vvp_io.c:194:vvp_prep_size()) [0x20000a811:0x9:0x0] updating i_size 2097152 00000080:00200000:1.0:1788572535.487854:0:69533:0:(vvp_io.c:893:vvp_io_read_start()) count:32768 iocb pos:32768 00010000:00010000:1.0:1788572535.487885:0:69533:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 4/1,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572535.487901:0:69533:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 4/1,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572535.487910:0:69533:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 7/2,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572535.487916:0:69533:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 7/2,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572535.487924:0:69533:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 6/2,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572535.497990:0:69533:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 3/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572535.498019:0:69533:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (1048576 1052671) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 3/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572535.498031:0:69533:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 6/1,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572535.498038:0:69533:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 6/1,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572535.498051:0:69533:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 5/1,0 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000008:00010000:1.0:1788572535.498314:0:69533:0:(osc_cache.c:3010:osc_queue_sync_pages()) ### no dlm lock: extent: ffff94d416fa1600 00010000:00010000:1.0:1788572535.498336:0:69533:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 6/2,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572535.498345:0:69533:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 5/1,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572535.498355:0:69533:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 5/1,0 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572535.498363:0:69533:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,0 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00000008:00010000:1.0:1788572535.498413:0:68272:0:(osc_cache.c:2022:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff94d416fa1600 00010000:00010000:1.0:1788572535.498445:0:68272:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (32768 36863) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 4/1,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000008:00100000:1.0:1788572535.498534:0:68272:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff94d4049bd180 - object 0x280000401:1794 offset 32768<>1048576 00000100:00100000:0.0:1788572535.498934:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1788572535.498957:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488524032:192.168.202.125@tcp:3:md5sum.0 00000008:00010000:0.0:1788572535.512154:0:2404:0:(osc_cache.c:910:osc_extent_finish()) ### no dlm lock: extent: ffff94d416fa1600 00000020:00200000:0.0:1788572535.512184:0:2404:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1788572535.512294:0:2404:0:(osc_cache.c:496:__osc_extent_remove()) ### no dlm lock: extent: ffff94d416fa1600 00000100:00100000:0.0:1788572535.512321:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488524032:192.168.202.125@tcp:3:md5sum.0 00000020:00010000:1.0:1788572535.512680:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff94d412304038 (ffff94d414ddaea8/0) at cl_lock_release():245 00000020:00010000:1.0:1788572535.512685:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d412304038 (ffff94d414ddaea8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788572535.512690:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d408c7ea30 (ffff94d414ddabd8/1) at cl_lock_cancel():142 00010000:00010000:1.0:1788572535.512703:0:69533:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 4/1,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572535.512719:0:69533:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00000020:00010000:1.0:1788572535.512738:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d412304038 (ffff94d414ddaea8/0) at cl_lock_fini():72 00000020:00010000:1.0:1788572535.512740:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d408c7ea30 (ffff94d414ddaea8/1) at cl_lock_fini():72 00000080:00200000:1.0:1788572535.512749:0:69533:0:(vvp_io.c:533:vvp_io_advance()) advancing 32768 bytes 00000080:00200000:1.0:1788572535.512791:0:69533:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x9:0x0] ignore/verify layout 0/0, layout version 3 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572535.512796:0:69533:0:(file.c:2238:ll_file_io_generic()) f132a.replay-single: 1 io complete with rc: 0, result: 32768, restart: 0 00000080:00200000:1.0:1788572535.512801:0:69533:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 32768 00000080:00200008:1.0:1788572535.512893:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 65536, count: 32768 00000080:00200000:1.0:1788572535.512919:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 16 00000080:00200000:1.0:1788572535.512926:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 17 00000080:00200000:1.0:1788572535.512930:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 18 00000080:00200000:1.0:1788572535.512933:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 19 00000080:00200000:1.0:1788572535.512937:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 20 00000080:00200000:1.0:1788572535.512940:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 21 00000080:00200000:1.0:1788572535.512942:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 22 00000080:00200000:1.0:1788572535.512945:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 23 00000080:00200008:1.0:1788572535.513016:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 98304, count: 32768 00000080:00200000:1.0:1788572535.513021:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 24 00000080:00200000:1.0:1788572535.513026:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 25 00000080:00200000:1.0:1788572535.513029:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 26 00000080:00200000:1.0:1788572535.513032:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 27 00000080:00200000:1.0:1788572535.513035:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 28 00000080:00200000:1.0:1788572535.513038:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 29 00000080:00200000:1.0:1788572535.513041:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 30 00000080:00200000:1.0:1788572535.513044:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 31 00000080:00200008:1.0:1788572535.513112:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 131072, count: 32768 00000080:00200000:1.0:1788572535.513115:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 32 00000080:00200000:1.0:1788572535.513119:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 33 00000080:00200000:1.0:1788572535.513122:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 34 00000080:00200000:1.0:1788572535.513125:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 35 00000080:00200000:1.0:1788572535.513129:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 36 00000080:00200000:1.0:1788572535.513133:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 37 00000080:00200000:1.0:1788572535.513136:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 38 00000080:00200000:1.0:1788572535.513139:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 39 00000080:00200008:1.0:1788572535.513238:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 163840, count: 32768 00000080:00200000:1.0:1788572535.513243:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 40 00000080:00200000:1.0:1788572535.513247:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 41 00000080:00200000:1.0:1788572535.513251:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 42 00000080:00200000:1.0:1788572535.513254:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 43 00000080:00200000:1.0:1788572535.513257:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 44 00000080:00200000:1.0:1788572535.513261:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 45 00000080:00200000:1.0:1788572535.513263:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 46 00000080:00200000:1.0:1788572535.513266:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 47 00000080:00200008:1.0:1788572535.513335:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 196608, count: 32768 00000080:00200000:1.0:1788572535.513339:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 48 00000080:00200000:1.0:1788572535.513343:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 49 00000080:00200000:1.0:1788572535.513346:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 50 00000080:00200000:1.0:1788572535.513350:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 51 00000080:00200000:1.0:1788572535.513353:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 52 00000080:00200000:1.0:1788572535.513357:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 53 00000080:00200000:1.0:1788572535.513359:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 54 00000080:00200000:1.0:1788572535.513362:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 55 00000080:00200008:1.0:1788572535.513431:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 229376, count: 32768 00000080:00200000:1.0:1788572535.513435:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 56 00000080:00200000:1.0:1788572535.513439:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 57 00000080:00200000:1.0:1788572535.513442:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 58 00000080:00200000:1.0:1788572535.513444:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 59 00000080:00200000:1.0:1788572535.513447:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 60 00000080:00200000:1.0:1788572535.513462:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 61 00000080:00200000:1.0:1788572535.513465:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 62 00000080:00200000:1.0:1788572535.513468:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 63 00000080:00200008:1.0:1788572535.513536:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 262144, count: 32768 00000080:00200000:1.0:1788572535.513541:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 64 00000080:00200000:1.0:1788572535.513545:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 65 00000080:00200000:1.0:1788572535.513549:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 66 00000080:00200000:1.0:1788572535.513552:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 67 00000080:00200000:1.0:1788572535.513555:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 68 00000080:00200000:1.0:1788572535.513558:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 69 00000080:00200000:1.0:1788572535.513560:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 70 00000080:00200000:1.0:1788572535.513563:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 71 00000080:00200008:1.0:1788572535.513633:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 294912, count: 32768 00000080:00200000:1.0:1788572535.513636:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 72 00000080:00200000:1.0:1788572535.513640:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 73 00000080:00200000:1.0:1788572535.513644:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 74 00000080:00200000:1.0:1788572535.513648:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 75 00000080:00200000:1.0:1788572535.513651:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 76 00000080:00200000:1.0:1788572535.513654:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 77 00000080:00200000:1.0:1788572535.513657:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 78 00000080:00200000:1.0:1788572535.513661:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 79 00000080:00200008:1.0:1788572535.513728:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 327680, count: 32768 00000080:00200000:1.0:1788572535.513733:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 80 00000080:00200000:1.0:1788572535.513736:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 81 00000080:00200000:1.0:1788572535.513739:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 82 00000080:00200000:1.0:1788572535.513743:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 83 00000080:00200000:1.0:1788572535.513745:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 84 00000080:00200000:1.0:1788572535.513749:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 85 00000080:00200000:1.0:1788572535.513752:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 86 00000080:00200000:1.0:1788572535.513755:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 87 00000080:00200008:1.0:1788572535.513823:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 360448, count: 32768 00000080:00200000:1.0:1788572535.513828:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 88 00000080:00200000:1.0:1788572535.513832:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 89 00000080:00200000:1.0:1788572535.513835:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 90 00000080:00200000:1.0:1788572535.513838:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 91 00000080:00200000:1.0:1788572535.513842:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 92 00000080:00200000:1.0:1788572535.513845:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 93 00000080:00200000:1.0:1788572535.513847:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 94 00000080:00200000:1.0:1788572535.513850:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 95 00000080:00200008:1.0:1788572535.513920:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 393216, count: 32768 00000080:00200000:1.0:1788572535.513924:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 96 00000080:00200000:1.0:1788572535.513927:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 97 00000080:00200000:1.0:1788572535.513931:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 98 00000080:00200000:1.0:1788572535.513933:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 99 00000080:00200000:1.0:1788572535.513937:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 100 00000080:00200000:1.0:1788572535.513940:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 101 00000080:00200000:1.0:1788572535.513943:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 102 00000080:00200000:1.0:1788572535.513947:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 103 00000080:00200008:1.0:1788572535.514014:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 425984, count: 32768 00000080:00200000:1.0:1788572535.514018:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 104 00000080:00200000:1.0:1788572535.514022:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 105 00000080:00200000:1.0:1788572535.514026:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 106 00000080:00200000:1.0:1788572535.514029:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 107 00000080:00200000:1.0:1788572535.514032:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 108 00000080:00200000:1.0:1788572535.514036:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 109 00000080:00200000:1.0:1788572535.514039:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 110 00000080:00200000:1.0:1788572535.514042:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 111 00000080:00200008:1.0:1788572535.514111:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 458752, count: 32768 00000080:00200000:1.0:1788572535.514115:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 112 00000080:00200000:1.0:1788572535.514119:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 113 00000080:00200000:1.0:1788572535.514122:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 114 00000080:00200000:1.0:1788572535.514126:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 115 00000080:00200000:1.0:1788572535.514129:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 116 00000080:00200000:1.0:1788572535.514132:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 117 00000080:00200000:1.0:1788572535.514135:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 118 00000080:00200000:1.0:1788572535.514138:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 119 00000080:00200008:1.0:1788572535.514236:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 491520, count: 32768 00000080:00200000:1.0:1788572535.514241:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 120 00000080:00200000:1.0:1788572535.514245:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 121 00000080:00200000:1.0:1788572535.514248:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 122 00000080:00200000:1.0:1788572535.514251:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 123 00000080:00200000:1.0:1788572535.514254:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 124 00000080:00200000:1.0:1788572535.514257:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 125 00000080:00200000:1.0:1788572535.514260:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 126 00000080:00200000:1.0:1788572535.514263:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 127 00000080:00200008:1.0:1788572535.514332:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 524288, count: 32768 00000080:00200000:1.0:1788572535.514336:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 128 00000080:00200000:1.0:1788572535.514339:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 129 00000080:00200000:1.0:1788572535.514342:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 130 00000080:00200000:1.0:1788572535.514345:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 131 00000080:00200000:1.0:1788572535.514348:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 132 00000080:00200000:1.0:1788572535.514351:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 133 00000080:00200000:1.0:1788572535.514355:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 134 00000080:00200000:1.0:1788572535.514358:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 135 00000080:00200008:1.0:1788572535.514426:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 557056, count: 32768 00000080:00200000:1.0:1788572535.514430:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 136 00000080:00200000:1.0:1788572535.514433:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 137 00000080:00200000:1.0:1788572535.514436:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 138 00000080:00200000:1.0:1788572535.514439:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 139 00000080:00200000:1.0:1788572535.514442:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 140 00000080:00200000:1.0:1788572535.514445:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 141 00000080:00200000:1.0:1788572535.514448:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 142 00000080:00200000:1.0:1788572535.514464:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 143 00000080:00200008:1.0:1788572535.514535:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 589824, count: 32768 00000080:00200000:1.0:1788572535.514539:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 144 00000080:00200000:1.0:1788572535.514543:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 145 00000080:00200000:1.0:1788572535.514546:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 146 00000080:00200000:1.0:1788572535.514549:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 147 00000080:00200000:1.0:1788572535.514553:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 148 00000080:00200000:1.0:1788572535.514556:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 149 00000080:00200000:1.0:1788572535.514559:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 150 00000080:00200000:1.0:1788572535.514562:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 151 00000080:00200008:1.0:1788572535.514630:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 622592, count: 32768 00000080:00200000:1.0:1788572535.514635:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 152 00000080:00200000:1.0:1788572535.514639:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 153 00000080:00200000:1.0:1788572535.514643:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 154 00000080:00200000:1.0:1788572535.514647:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 155 00000080:00200000:1.0:1788572535.514651:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 156 00000080:00200000:1.0:1788572535.514655:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 157 00000080:00200000:1.0:1788572535.514658:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 158 00000080:00200000:1.0:1788572535.514662:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 159 00000080:00200008:1.0:1788572535.514731:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 655360, count: 32768 00000080:00200000:1.0:1788572535.514736:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 160 00000080:00200000:1.0:1788572535.514740:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 161 00000080:00200000:1.0:1788572535.514743:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 162 00000080:00200000:1.0:1788572535.514746:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 163 00000080:00200000:1.0:1788572535.514749:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 164 00000080:00200000:1.0:1788572535.514752:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 165 00000080:00200000:1.0:1788572535.514755:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 166 00000080:00200000:1.0:1788572535.514758:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 167 00000080:00200008:1.0:1788572535.514826:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 688128, count: 32768 00000080:00200000:1.0:1788572535.514830:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 168 00000080:00200000:1.0:1788572535.514834:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 169 00000080:00200000:1.0:1788572535.514837:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 170 00000080:00200000:1.0:1788572535.514840:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 171 00000080:00200000:1.0:1788572535.514843:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 172 00000080:00200000:1.0:1788572535.514846:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 173 00000080:00200000:1.0:1788572535.514850:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 174 00000080:00200000:1.0:1788572535.514853:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 175 00000080:00200008:1.0:1788572535.514921:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 720896, count: 32768 00000080:00200000:1.0:1788572535.514926:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 176 00000080:00200000:1.0:1788572535.514930:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 177 00000080:00200000:1.0:1788572535.514933:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 178 00000080:00200000:1.0:1788572535.514936:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 179 00000080:00200000:1.0:1788572535.514939:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 180 00000080:00200000:1.0:1788572535.514943:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 181 00000080:00200000:1.0:1788572535.514946:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 182 00000080:00200000:1.0:1788572535.514949:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 183 00000080:00200008:1.0:1788572535.515016:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 753664, count: 32768 00000080:00200000:1.0:1788572535.515020:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 184 00000080:00200000:1.0:1788572535.515023:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 185 00000080:00200000:1.0:1788572535.515026:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 186 00000080:00200000:1.0:1788572535.515029:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 187 00000080:00200000:1.0:1788572535.515032:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 188 00000080:00200000:1.0:1788572535.515035:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 189 00000080:00200000:1.0:1788572535.515039:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 190 00000080:00200000:1.0:1788572535.515042:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 191 00000080:00200008:1.0:1788572535.515110:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 786432, count: 32768 00000080:00200000:1.0:1788572535.515115:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 192 00000080:00200000:1.0:1788572535.515119:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 193 00000080:00200000:1.0:1788572535.515122:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 194 00000080:00200000:1.0:1788572535.515125:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 195 00000080:00200000:1.0:1788572535.515129:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 196 00000080:00200000:1.0:1788572535.515132:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 197 00000080:00200000:1.0:1788572535.515135:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 198 00000080:00200000:1.0:1788572535.515139:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 199 00000080:00200008:1.0:1788572535.515238:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 819200, count: 32768 00000080:00200000:1.0:1788572535.515243:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 200 00000080:00200000:1.0:1788572535.515247:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 201 00000080:00200000:1.0:1788572535.515250:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 202 00000080:00200000:1.0:1788572535.515254:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 203 00000080:00200000:1.0:1788572535.515257:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 204 00000080:00200000:1.0:1788572535.515260:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 205 00000080:00200000:1.0:1788572535.515263:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 206 00000080:00200000:1.0:1788572535.515265:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 207 00000080:00200008:1.0:1788572535.515333:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 851968, count: 32768 00000080:00200000:1.0:1788572535.515338:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 208 00000080:00200000:1.0:1788572535.515341:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 209 00000080:00200000:1.0:1788572535.515345:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 210 00000080:00200000:1.0:1788572535.515347:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 211 00000080:00200000:1.0:1788572535.515350:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 212 00000080:00200000:1.0:1788572535.515353:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 213 00000080:00200000:1.0:1788572535.515356:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 214 00000080:00200000:1.0:1788572535.515359:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 215 00000080:00200008:1.0:1788572535.515428:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 884736, count: 32768 00000080:00200000:1.0:1788572535.515433:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 216 00000080:00200000:1.0:1788572535.515438:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 217 00000080:00200000:1.0:1788572535.515441:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 218 00000080:00200000:1.0:1788572535.515445:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 219 00000080:00200000:1.0:1788572535.517647:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 220 00000080:00200000:2.0:1788572535.523804:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 221 00000080:00200000:2.0:1788572535.523824:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 222 00000080:00200000:2.0:1788572535.523830:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 223 00000080:00200008:2.0:1788572535.524046:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 917504, count: 32768 00000080:00200000:2.0:1788572535.524055:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 224 00000080:00200000:2.0:1788572535.524060:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 225 00000080:00200000:2.0:1788572535.524064:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 226 00000080:00200000:2.0:1788572535.524067:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 227 00000080:00200000:2.0:1788572535.524070:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 228 00000080:00200000:2.0:1788572535.524074:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 229 00000080:00200000:2.0:1788572535.524077:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 230 00000080:00200000:2.0:1788572535.524080:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 231 00000080:00200008:2.0:1788572535.524149:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 950272, count: 32768 00000080:00200000:2.0:1788572535.524154:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 232 00000080:00200000:2.0:1788572535.524158:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 233 00000080:00200000:2.0:1788572535.524162:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 234 00000080:00200000:2.0:1788572535.524165:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 235 00000080:00200000:2.0:1788572535.524169:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 236 00000080:00200000:2.0:1788572535.524171:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 237 00000080:00200000:2.0:1788572535.524174:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 238 00000080:00200000:2.0:1788572535.524178:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 239 00000080:00200008:2.0:1788572535.524276:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 983040, count: 32768 00000080:00200000:2.0:1788572535.524281:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 240 00000080:00200000:2.0:1788572535.524285:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 241 00000080:00200000:2.0:1788572535.524289:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 242 00000080:00200000:2.0:1788572535.524300:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 243 00000080:00200000:2.0:1788572535.524305:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 244 00000080:00200000:2.0:1788572535.524308:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 245 00000080:00200000:2.0:1788572535.524311:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 246 00000080:00200000:2.0:1788572535.524314:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 247 00000080:00200008:2.0:1788572535.524383:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1015808, count: 32768 00000080:00200000:2.0:1788572535.524387:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 248 00000080:00200000:2.0:1788572535.524391:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 249 00000080:00200000:2.0:1788572535.524395:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 250 00000080:00200000:2.0:1788572535.524398:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 251 00000080:00200000:2.0:1788572535.524402:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 252 00000080:00200000:2.0:1788572535.524406:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 253 00000080:00200000:2.0:1788572535.524410:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 254 00000080:00200000:2.0:1788572535.524413:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 255 00000080:00200008:2.0:1788572535.524491:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1048576, count: 32768 00000080:00200008:2.0:1788572535.524576:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1081344, count: 32768 00000080:00200008:2.0:1788572535.524659:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1114112, count: 32768 00000080:00200008:2.0:1788572535.524741:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1146880, count: 32768 00000080:00200008:2.0:1788572535.524822:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1179648, count: 32768 00000080:00200008:2.0:1788572535.525478:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1212416, count: 32768 00000080:00200008:2.0:1788572535.525567:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1245184, count: 32768 00000080:00200008:2.0:1788572535.525649:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1277952, count: 32768 00000080:00200008:2.0:1788572535.525741:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1310720, count: 32768 00000080:00200008:2.0:1788572535.525822:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1343488, count: 32768 00000080:00200008:2.0:1788572535.527139:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1376256, count: 32768 00000080:00200008:2.0:1788572535.527253:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1409024, count: 32768 00000080:00200008:2.0:1788572535.527412:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1441792, count: 32768 00000080:00200008:2.0:1788572535.527506:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1474560, count: 32768 00000080:00200008:2.0:1788572535.527588:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1507328, count: 32768 00000080:00200008:2.0:1788572535.527669:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1540096, count: 32768 00000080:00200008:2.0:1788572535.527752:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1572864, count: 32768 00000080:00200008:2.0:1788572535.527834:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1605632, count: 32768 00000080:00200008:2.0:1788572535.532486:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1638400, count: 32768 00000080:00200008:2.0:1788572535.532596:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1671168, count: 32768 00000080:00200008:2.0:1788572535.532677:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1703936, count: 32768 00000080:00200008:2.0:1788572535.532759:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1736704, count: 32768 00000080:00200008:2.0:1788572535.532842:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1769472, count: 32768 00000080:00200008:2.0:1788572535.532922:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1802240, count: 32768 00000080:00200008:2.0:1788572535.533006:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1835008, count: 32768 00000080:00200008:2.0:1788572535.533090:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1867776, count: 32768 00000080:00200008:2.0:1788572535.533173:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1900544, count: 32768 00000080:00200008:2.0:1788572535.533279:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1933312, count: 32768 00000080:00200008:3.0:1788572535.537512:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1966080, count: 32768 00000080:00200008:3.0:1788572535.537608:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1998848, count: 32768 00000080:00200008:3.0:1788572535.537691:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 2031616, count: 32768 00000080:00200008:3.0:1788572535.537769:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 2064384, count: 32768 00000080:00200008:3.0:1788572535.537850:0:69533:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 2097152, count: 32768 00000080:00200000:3.0:1788572535.537889:0:69533:0:(rw.c:1925:ll_readpage()) fast read pgno: 512 00000080:00200000:3.0:1788572535.537934:0:69533:0:(file.c:2090:ll_file_io_generic()) f132a.replay-single: read ppos: 2097152, bytes: 32768 00000080:00200000:3.0:1788572535.537939:0:69533:0:(file.c:1848:ll_io_set_mirror()) f132a.replay-single: desiginated mirror: 0 00000020:00200000:3.0:1788572535.537949:0:69533:0:(cl_io.c:214:cl_io_rw_init()) header@0000000099bd4c4d[0x110, 513, [0x20000a811:0x9:0x0] hash] 00000020:00200000:3.0:1788572535.537952:0:69533:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [2097152, 2129920) 0 0 00000080:00200000:3.0:1788572535.537958:0:69533:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572535.537980:0:69533:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [2097152, 2129920) 2129920, 32768 00020000:00200000:3.0:1788572535.537985:0:69533:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:3.0:1788572535.538016:0:69533:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [512, 519] 00000008:00010000:3.0:1788572535.538033:0:69533:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1788572535.538039:0:69533:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff94d4129db4b0, osc lock ffff94d408938260, flags 0x0 00010000:00010000:3.0:1788572535.538063:0:69533:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 3/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788572535.538081:0:69533:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (2097152 2129919) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 3/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788572535.538094:0:69533:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 7/1,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788572535.538104:0:69533:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 7/1,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788572535.538114:0:69533:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 6/1,0 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00000080:00200000:3.0:1788572535.538123:0:69533:0:(vvp_io.c:835:vvp_io_read_start()) f132a.replay-single: read [2097152, 2129920) 00000001:00010000:3.0:1788572535.538138:0:69533:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000a811:0x9:0x0] 00000008:00010000:3.0:1788572535.538159:0:69533:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1788572535.538161:0:69533:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff94d411bd1830, osc lock ffff94d408938850, flags 0x400001002 00000008:00010000:3.0:1788572535.538165:0:69533:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1788572535.538167:0:69533:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff94d411bd1870, osc lock ffff94d408938f70, flags 0x400001002 00000008:00010000:3.0:1788572535.539252:0:69533:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1788572535.539263:0:69533:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff94d411bd18b0, osc lock ffff94d408938720, flags 0x400001002 00010000:00010000:3.0:1788572535.539433:0:69533:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788572535.539462:0:69533:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 1048575) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00000020:00200000:3.0:1788572535.539475:0:69533:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1788572535.539479:0:69533:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788572535.539488:0:69533:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1788572535.539514:0:69533:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 5/1,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788572535.539522:0:69533:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (1048576 18446744073709551615) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 5/1,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00000020:00200000:3.0:1788572535.539534:0:69533:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1788572535.539537:0:69533:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 8/1,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788572535.539544:0:69533:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 7/1,0 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788572535.539556:0:69533:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x703:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a08 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572535.539563:0:69533:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x703:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a08 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:3.0:1788572535.539570:0:69533:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1788572535.539573:0:69533:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x703:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a08 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572535.539580:0:69533:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x703:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a08 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00200000:3.0:1788572535.539588:0:69533:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1788572535.539597:0:69533:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000a811:0x9:0x0] updating i_size 2097152 i_blocks 0 00000020:00010000:3.0:1788572535.539604:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff94d405c4cb90 (ffff94d40625f758/0) at cl_lock_release():245 00000020:00010000:3.0:1788572535.539608:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d405c4cb90 (ffff94d40625f758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788572535.539611:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d411bd1830 (ffff94d40625f518/1) at cl_lock_cancel():142 00000020:00010000:3.0:1788572535.539615:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d411bd1870 (ffff94d40625fea8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1788572535.539618:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d411bd18b0 (ffff94d40625f128/1) at cl_lock_cancel():142 00000020:00010000:3.0:1788572535.539620:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d405c4cb90 (ffff94d40625f758/0) at cl_lock_fini():72 00000020:00010000:3.0:1788572535.539624:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d411bd1830 (ffff94d40625f758/1) at cl_lock_fini():72 00000020:00010000:3.0:1788572535.539630:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d411bd1870 (ffff94d40625f758/1) at cl_lock_fini():72 00000020:00010000:3.0:1788572535.539633:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d411bd18b0 (ffff94d40625f758/1) at cl_lock_fini():72 00000020:00010000:3.0:1788572535.539647:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff94d4087323f8 (ffff94d40625f758/0) at cl_lock_release():245 00000020:00010000:3.0:1788572535.539650:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d4087323f8 (ffff94d40625f758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788572535.539652:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d4129db4b0 (ffff94d40625fea8/1) at cl_lock_cancel():142 00010000:00010000:3.0:1788572535.539656:0:69533:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 5/1,0 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788572535.539664:0:69533:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,0 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00000020:00010000:3.0:1788572535.539674:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d4087323f8 (ffff94d40625f758/0) at cl_lock_fini():72 00000020:00010000:3.0:1788572535.539676:0:69533:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d4129db4b0 (ffff94d40625f758/1) at cl_lock_fini():72 00000080:00200000:3.0:1788572535.539682:0:69533:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000080:00200000:3.0:1788572535.539729:0:69533:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x9:0x0] ignore/verify layout 0/0, layout version 3 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572535.539734:0:69533:0:(file.c:2238:ll_file_io_generic()) f132a.replay-single: 1 io complete with rc: 0, result: 0, restart: 0 00000080:00200000:3.0:1788572535.539737:0:69533:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 0 00000080:00200008:3.0:1788572535.539763:0:69533:0:(file.c:5203:ll_file_seek()) START file f132a.replay-single:[0x20000a811:0x9:0x0], offset: 0, type: SEEK_CUR 00000080:00200008:3.0:1788572535.539772:0:69533:0:(file.c:5234:ll_file_seek()) COMPLETED file f132a.replay-single:[0x20000a811:0x9:0x0], offset: 0, type: SEEK_CUR, rc = 2097152 00000080:00200008:3.0:1788572535.539789:0:69533:0:(file.c:479:ll_file_release()) START file f132a.replay-single:[0x20000a811:0x9:0x0](ffff94d410bc3248), flags 100000 00000002:00100000:3.0:1788572535.539869:0:69533:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff94d408c55180 x1875450488523776/t107374182415(107374182415) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 800/872 e 0 to 0 dl 1788572551 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'md5sum.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788572535.539886:0:69533:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788572535.539892:0:69533:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c56680 x1875450488524288/t0(0) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/760 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788572535.540104:0:69533:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job md5sum:9bf287c0-f582-4b83-84c1-15e766218620:69533:1875450488524288:192.168.202.125@tcp:35:md5sum.0 00000100:00100000:3.0:1788572535.540200:0:69533:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4054110c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572535.546786:0:69533:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job md5sum:9bf287c0-f582-4b83-84c1-15e766218620:69533:1875450488524288:192.168.202.125@tcp:35:md5sum.0 00000002:00100000:1.0:1788572535.546824:0:69533:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff94d408c55180 x1875450488523776/t107374182415(107374182415) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 800/872 e 0 to 0 dl 1788572551 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'md5sum.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572535.588941:0:69535:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572535.588976:0:69535:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572535.588996:0:69535:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167400/0x1e260726fbbc0b46 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0a16 expref: -99 pid: 69531 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572535.589012:0:69535:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167400/0x1e260726fbbc0b46 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0a16 expref: -99 pid: 69531 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788572535.589020:0:69535:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572535.591529:0:69535:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572535.591548:0:69535:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0b46 00000080:00010000:1.0:1788572535.591562:0:69535:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b46 from it ffffb68a4209fbf8 00010000:00010000:1.0:1788572535.591569:0:69535:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167400/0x1e260726fbbc0b46 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0a16 expref: -99 pid: 69531 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572535.591584:0:69535:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167400/0x1e260726fbbc0b46 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0a16 expref: -99 pid: 69531 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788572535.591605:0:69535:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572535.591618:0:69535:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f132a.replay-single, flags=267 00000080:00200000:1.0:1788572535.591706:0:69535:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572535.591730:0:69535:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572535.591746:0:69535:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167400/0x1e260726fbbc0b46 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0a16 expref: -99 pid: 69531 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572535.591758:0:69535:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167400/0x1e260726fbbc0b46 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0a16 expref: -99 pid: 69531 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788572535.591766:0:69535:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572535.591774:0:69535:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572535.591776:0:69535:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0b46 00000080:00010000:1.0:1788572535.591780:0:69535:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b46 from it ffffb68a4209fbb0 00010000:00010000:1.0:1788572535.591782:0:69535:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167400/0x1e260726fbbc0b46 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0a16 expref: -99 pid: 69531 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572535.591789:0:69535:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167400/0x1e260726fbbc0b46 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0a16 expref: -99 pid: 69531 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1788572535.591796:0:69535:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 44 00000080:00200008:1.0:1788572535.591805:0:69535:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 304000 00000080:00200080:1.0:1788572535.591837:0:69535:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c0086915 arg=19843d0 00000100:00100000:1.0:1788572535.591909:0:69535:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69535:1875450488524416:192.168.202.125@tcp:34:lfs.0 00000100:00100000:1.0:1788572535.592005:0:69535:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408c8b6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572535.595484:0:69535:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69535:1875450488524416:192.168.202.125@tcp:34:lfs.0 00000080:00200008:1.0:1788572535.595687:0:69535:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 304000 00010000:00010000:1.0:1788572536.453551:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572537.477899:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1788572537.541816:0:51031:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572537.542032:0:51031:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:0.0:1788572537.542059:0:51031:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x0 00000080:00200000:0.0:1788572537.542101:0:51031:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572537.542134:0:51031:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572537.542139:0:51031:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:0.0:1788572537.542149:0:51031:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x10 00000008:00010000:0.0:1788572537.542186:0:51031:0:(osc_cache.c:2022:try_to_add_extent_for_io()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00000008:00010000:0.0:1788572537.542244:0:51031:0:(osc_cache.c:1176:osc_extent_make_ready()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00010000:00010000:0.0:1788572537.544016:0:51031:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (1048576 1052671) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00000008:00100000:0.0:1788572537.545001:0:51031:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff94d408d3df80 - object 0x2c0000401:1794 offset 1048576<>2097152 00000100:00100000:0.0:1788572537.545117:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488524672:192.168.202.125@tcp:16:kworker.0 00000100:00100000:1.0:1788572537.545306:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488524544:192.168.202.125@tcp:4:md5sum.0 00000100:00100000:0.0:1788572537.618161:0:2403:0:(client.c:3112:ptlrpc_free_committed()) lustre-OST0001-osc-ffff94d405cc3800: committing for last_committed 4294968862 gen 17 00000100:00100000:0.0:1788572537.618238:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488524672:192.168.202.125@tcp:16:kworker.0 00000080:00200000:3.0:1788572537.618388:0:51031:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000008:00010000:1.0:1788572537.619349:0:2404:0:(osc_cache.c:910:osc_extent_finish()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00000008:00010000:1.0:1788572537.619840:0:2404:0:(osc_cache.c:496:__osc_extent_remove()) ### extent: 00000000329e1861 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00000100:00100000:1.0:1788572537.619873:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488524544:192.168.202.125@tcp:4:md5sum.0 00000100:00100000:3.0:1788572538.056195:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488524800:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572538.056340:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488525056:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572538.057471:0:2405:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1788572538.057486:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488524928:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572538.057522:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488525184:192.168.202.125@tcp:400:kworker.0 00000100:00080000:3.0:1788572538.063448:0:2405:0:(recover.c:205:ptlrpc_request_handle_notconn()) import lustre-MDT0000-mdc-ffff94d405cc3800 of lustre-MDT0000_UUID@192.168.202.125@tcp abruptly disconnected: reconnecting 00000100:00080000:3.0:1788572538.063456:0:2405:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:3.0:1788572538.063463:0:2405:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff94d405cc3800: Connection to lustre-MDT0000 (at 192.168.202.125@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:3.0:1788572538.063471:0:2405:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572538.063477:0:2405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3650 00000100:00080000:3.0:1788572538.063489:0:2405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572538.063502:0:2405:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572538.063534:0:2405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488525312/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572538.063575:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488524928:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572538.063596:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488525184:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572538.063607:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488525056:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572538.063773:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488525312:192.168.202.125@tcp:38:ptlrpcd_01_00.0 00000100:00080000:1.0:1788572538.073735:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572538.073761:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-19) 00000100:00100000:1.0:1788572538.073767:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488525312:192.168.202.125@tcp:38:ptlrpcd_01_00.0 00010000:00010000:0.0:1788572538.503789:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572539.525708:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572540.549940:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572541.573658:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572542.598570:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572543.173629:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572543.173636:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572543.173640:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572543.173659:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3691 00000100:00080000:2.0:1788572543.173670:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: tried all connections, increasing latency to 5s 00000100:00080000:2.0:1788572543.173678:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572543.173688:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572543.173706:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63100 x1875450488525568/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572543.173799:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488525568:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788572543.179602:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488525440:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572543.179711:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488525824:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572543.179747:0:2405:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:2.0:1788572543.179754:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488525696:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572543.179779:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488525952:192.168.202.125@tcp:400:kworker.0 00000100:00080000:1.0:1788572543.186019:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572543.186034:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-19) 00000100:00100000:1.0:1788572543.186039:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488525568:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788572543.186975:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488525696:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572543.186992:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488525952:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572543.187001:0:2405:0:(client.c:3112:ptlrpc_free_committed()) lustre-OST0000-osc-ffff94d405cc3800: committing for last_committed 4294968865 gen 16 00000100:00100000:2.0:1788572543.187022:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488525824:192.168.202.125@tcp:400:kworker.0 00010000:00010000:0.0:1788572543.621507:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572544.647008:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572545.669537:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572546.693362:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572547.717352:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572548.293453:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572548.293461:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572548.293464:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572548.293476:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3696 00000100:00080000:2.0:1788572548.293498:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: tried all connections, increasing latency to 10s 00000100:00080000:2.0:1788572548.293506:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788572548.293517:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788572548.293540:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488526080:192.168.202.125@tcp:400:kworker.0 00000008:00100000:2.0:1788572548.293644:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572548.293665:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488526208/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572548.293715:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488526336:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572548.293767:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788572548.293774:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488526464:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572548.293818:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488526592:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572548.294137:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488526208:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788572548.300325:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488526336:192.168.202.125@tcp:400:kworker.0 00000100:00080000:1.0:1788572548.308244:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572548.308261:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-19) 00000100:00100000:1.0:1788572548.308266:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488526208:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788572548.309660:0:2406:0:(client.c:3112:ptlrpc_free_committed()) lustre-OST0001-osc-ffff94d405cc3800: committing for last_committed 4294968863 gen 17 00000100:00100000:2.0:1788572548.309702:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488526592:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572548.309717:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488526464:192.168.202.125@tcp:400:kworker.0 00010000:00010000:1.0:1788572548.741493:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572549.765363:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572550.802080:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572551.813511:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572552.837505:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572553.414080:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488526720:192.168.202.125@tcp:400:kworker.0 00000100:00080000:0.0:1788572553.414849:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572553.414856:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572553.414860:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572553.414867:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3701 00000100:00080000:0.0:1788572553.414875:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: tried all connections, increasing latency to 15s 00000100:00080000:0.0:1788572553.414883:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572553.414894:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572553.414917:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488526848/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572553.415058:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488526848:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788572553.423842:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572553.423857:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-19) 00000100:00100000:3.0:1788572553.423863:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488526848:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572553.427155:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788572553.427197:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488526976:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572553.428671:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488527232:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572553.428746:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788572553.428752:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488527104:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572553.429555:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488526976:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572553.448919:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488527104:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572553.448937:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488527232:192.168.202.125@tcp:400:kworker.0 00010000:00010000:2.0:1788572553.866651:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:3.0:1788572554.373550:0:2405:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572538/real 1788572538] req@ffff94d408c56680 x1875450488524800/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572554 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572554.373587:0:2405:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff94d408c56680 x1875450488524800/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572554 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572554.373605:0:2405:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from FULL to DISCONN 00000100:02020000:3.0:1788572554.373609:0:2405:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.202.125@tcp: Connection to MGS (at 192.168.202.125@tcp) was lost; in progress operations using this service will fail 00000100:00080000:3.0:1788572554.373612:0:2405:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.202.125@tcp for MGC192.168.202.125@tcp not replayable, auto-deactivating 00000100:00080000:3.0:1788572554.373614:0:2405:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:3.0:1788572554.373618:0:2405:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff94d408c56680 x1875450488524800/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572554 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572554.373626:0:2405:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff94d408c62a00 x1875450488525440/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572559 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572554.373638:0:2405:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff94d408c63b80 x1875450488526080/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572564 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572554.373664:0:2405:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff94d408d3ce00 x1875450488526720/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572569 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:3.0:1788572554.373683:0:2405:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:3.0:1788572554.373686:0:2405:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:3.0:1788572554.373688:0:2405:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:3.0:1788572554.373689:0:2405:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00100000:3.0:1788572554.373725:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488525440:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572554.373731:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2405:1875450488524800:192.168.202.125@tcp:400:kworker.0 00000100:00080000:0.0:1788572554.373828:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) e60431de-58b9-4f69-9048-11a3d82ddcdd->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572554.373833:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572554.373837:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572554.373842:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3648 00000100:00080000:0.0:1788572554.373857:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d40490a800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788572554.373892:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488527360/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572554.373928:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572554.373932:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572554.373933:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572554.373937:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3706 00000100:00080000:0.0:1788572554.373941:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572554.373945:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572554.373954:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572554.373967:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488527488/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572554.374179:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e60431de-58b9-4f69-9048-11a3d82ddcdd:2402:1875450488527360:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788572554.374283:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488527488:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572554.374846:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488527616:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572554.374901:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488527872:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572554.374945:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488526720:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572554.374964:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788572554.374969:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488527744:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572554.375168:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488526080:192.168.202.125@tcp:400:kworker.0 00000100:00080000:0.0:1788572554.379169:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572554.379183:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-19) 00000100:00100000:0.0:1788572554.379188:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488527488:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572554.382681:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488527744:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572554.382701:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488527616:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572554.382714:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488527872:192.168.202.125@tcp:400:kworker.0 00010000:00010000:1.0:1788572554.885448:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572555.909398:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572556.934464:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572557.957459:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572558.981360:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572559.557447:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) e60431de-58b9-4f69-9048-11a3d82ddcdd->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572559.557458:0:66171:0:(pinger.c:247:ptlrpc_pinger_process_import()) e60431de-58b9-4f69-9048-11a3d82ddcdd->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788572559.557461:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572559.557465:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572559.557469:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572559.557480:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3707 00000100:00080000:2.0:1788572559.557491:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572559.557499:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572559.557510:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572559.557546:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488528000/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572559.557633:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488528256:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572559.557708:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:2.0:1788572559.557714:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488528128:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572559.557738:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488528384:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572559.559586:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488528000:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788572559.559768:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488528384:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572559.559783:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488528256:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572559.567177:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488528128:192.168.202.125@tcp:400:kworker.0 00000100:00080000:0.0:1788572559.571583:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572559.571599:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-19) 00000100:00100000:0.0:1788572559.571604:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488528000:192.168.202.125@tcp:38:kworker.0 00010000:00010000:3.0:1788572560.011421:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572561.029388:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572562.053345:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572563.077741:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572564.101471:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572564.677703:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572554/real 1788572554] req@ffff94d408d3d880 x1875450488527360/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572564 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572564.677734:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488527360/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572564 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572564.677752:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572564.677774:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572564.677778:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e60431de-58b9-4f69-9048-11a3d82ddcdd:2402:1875450488527360:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788572564.677843:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) e60431de-58b9-4f69-9048-11a3d82ddcdd->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572564.677848:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572564.677851:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572564.677857:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3707 00000100:00080000:1.0:1788572564.677868:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 5s 00000100:00080000:1.0:1788572564.677874:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d40490a800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788572564.677903:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488528512/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572564.677929:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572564.677942:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572564.677944:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572564.677948:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3712 00000100:00080000:1.0:1788572564.677953:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572564.677957:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572564.677966:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572564.677979:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdf80 x1875450488528640/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572564.678033:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488528768:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572564.678112:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488529024:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572564.678140:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788572564.678145:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488528896:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572564.678425:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e60431de-58b9-4f69-9048-11a3d82ddcdd:2402:1875450488528512:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788572564.678463:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488528640:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572564.682267:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488528768:192.168.202.125@tcp:400:kworker.0 00000100:00080000:0.0:1788572564.682971:0:2402:0:(import.c:1132:ptlrpc_connect_interpret()) MGC192.168.202.125@tcp: connect to target with instance 0 10000000:01000000:0.0:1788572564.682986:0:2402:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:02000400:0.0:1788572564.682992:0:2402:0:(import.c:1284:ptlrpc_connect_interpret()) Evicted from MGS (at 192.168.202.125@tcp) after server handle changed from 0xec901abfeb4e0490 to 0xec901abfeb4e0a9b 00000100:00080000:0.0:1788572564.682996:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from CONNECTING to EVICTED 00000100:00080000:0.0:1788572564.683001:0:2402:0:(import.c:1674:ptlrpc_import_recovery_state_machine()) evicted from MGS@192.168.202.125@tcp; invalidating 00000100:00100000:0.0:1788572564.683546:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e60431de-58b9-4f69-9048-11a3d82ddcdd:2402:1875450488528512:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788572564.683567:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572564.683571:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-19) 00000100:00100000:0.0:1788572564.683575:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488528640:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788572564.683642:0:69759:0:(import.c:1598:ptlrpc_invalidate_import_thread()) thread invalidate import MGC192.168.202.125@tcp to MGS@192.168.202.125@tcp 00000100:00100000:3.0:1788572564.683647:0:69759:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:3.0:1788572564.683649:0:69759:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00010000:00010000:3.0:1788572564.683663:0:69759:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.125@tcp lock: ffff94d412167e00/0x1e260726fbbc0aac lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110e400000000 nid: local remote: 0xec901abfeb4e05bd expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788572564.683689:0:69759:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.125@tcp lock: ffff94d412167e00/0x1e260726fbbc0aac lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110f400000000 nid: local remote: 0xec901abfeb4e05bd expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572564.683696:0:69759:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:3.0:1788572564.683712:0:69759:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572564.683723:0:69759:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.125@tcp lock: ffff94d412167e00/0x1e260726fbbc0aac lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x590f400000000 nid: local remote: 0xec901abfeb4e05bd expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572564.683736:0:69759:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.125@tcp lock: ffff94d412167800/0x1e260726fbbc0aba lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0xec901abfeb4e05cb expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788572564.683743:0:69759:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.125@tcp lock: ffff94d412167800/0x1e260726fbbc0aba lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0xec901abfeb4e05cb expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572564.683748:0:69759:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:3.0:1788572564.683752:0:69759:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572564.683755:0:69759:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.125@tcp lock: ffff94d412167800/0x1e260726fbbc0aba lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0xec901abfeb4e05cb expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572564.683768:0:69759:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.125@tcp lock: ffff94d412537000/0x1e260726fbbc0aa5 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0xec901abfeb4e05b6 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788572564.683775:0:69759:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.125@tcp lock: ffff94d412537000/0x1e260726fbbc0aa5 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0xec901abfeb4e05b6 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572564.683781:0:69759:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:3.0:1788572564.683785:0:69759:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572564.683788:0:69759:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.125@tcp lock: ffff94d412537000/0x1e260726fbbc0aa5 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0xec901abfeb4e05b6 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572564.683801:0:69759:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.125@tcp lock: ffff94d412536000/0x1e260726fbbc0ab3 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0xec901abfeb4e05c4 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788572564.683808:0:69759:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.125@tcp lock: ffff94d412536000/0x1e260726fbbc0ab3 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0xec901abfeb4e05c4 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572564.683812:0:69759:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 00010000:00010000:3.0:1788572564.683815:0:69759:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572564.683819:0:69759:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.125@tcp lock: ffff94d412536000/0x1e260726fbbc0ab3 lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0xec901abfeb4e05c4 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:3.0:1788572564.683832:0:69759:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from EVICTED to RECOVER 00000100:00080000:3.0:1788572564.684101:0:69759:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from RECOVER to FULL 10000000:01000000:3.0:1788572564.684104:0:69759:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:3.0:1788572564.684117:0:69759:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:02000000:3.0:1788572564.684121:0:69759:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) MGC192.168.202.125@tcp: Connection restored to 192.168.202.125@tcp (at 192.168.202.125@tcp) 00000100:00100000:1.0:1788572564.686132:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488528896:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572564.686167:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488529024:192.168.202.125@tcp:400:kworker.0 00010000:00010000:0.0:1788572565.126027:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788572565.893313:0:42451:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788572565.893322:0:42451:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-cliir-ffff94d405cc3800 from 113 10000000:01000000:0.0:1788572565.893329:0:42451:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788572565.893384:0:42451:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412167200/0x1e260726fbbc0b5b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572565.893397:0:42451:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.125@tcp lock: ffff94d412167200/0x1e260726fbbc0b5b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572565.893409:0:42451:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.125@tcp lock: ffff94d412167200/0x1e260726fbbc0b5b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572565.893440:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488529152:192.168.202.125@tcp:101:ll_cfg_requeue.0 00000100:00100000:0.0:1788572565.893552:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416e1f300 going to sleep for 11 seconds 00000100:00100000:0.0:1788572565.896547:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488529152:192.168.202.125@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788572565.896576:0:42451:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.125@tcp lock: ffff94d412167200/0x1e260726fbbc0b5b lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0aa2 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572565.896588:0:42451:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.125@tcp lock: ffff94d412167200/0x1e260726fbbc0b5b lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xec901abfeb4e0aa2 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572565.896680:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488529280:192.168.202.125@tcp:256:ll_cfg_requeue.0 00000100:00100000:0.0:1788572565.896777:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416e1f300 going to sleep for 11 seconds 00000100:00100000:0.0:1788572565.899183:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488529280:192.168.202.125@tcp:256:ll_cfg_requeue.0 10000000:01000000:0.0:1788572565.899256:0:42451:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.202.125@tcp: configuration from log 'lustre-cliir' failed (-2). 00010000:00010000:0.0:1788572565.899267:0:42451:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412167200/0x1e260726fbbc0b5b lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0aa2 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572565.899278:0:42451:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.125@tcp lock: ffff94d412167200/0x1e260726fbbc0b5b lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0aa2 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00000400:0.0:1788572565.899289:0:42451:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.125@tcp: IR log lustre-cliir failed, not fatal: rc = -2 10000000:01000000:0.0:1788572565.906837:0:42451:0:(mgc_request.c:611:do_requeue()) updating log lustre-client 10000000:01000000:0.0:1788572565.906845:0:42451:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-client-ffff94d405cc3800 from 52 10000000:01000000:0.0:1788572565.906851:0:42451:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:0.0:1788572565.906932:0:42451:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412166600/0x1e260726fbbc0b62 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572565.906944:0:42451:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.125@tcp lock: ffff94d412166600/0x1e260726fbbc0b62 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572565.906955:0:42451:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.125@tcp lock: ffff94d412166600/0x1e260726fbbc0b62 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572565.906984:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488529408:192.168.202.125@tcp:101:ll_cfg_requeue.0 00000100:00100000:0.0:1788572565.907159:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416e1f300 going to sleep for 11 seconds 00000100:00100000:1.0:1788572565.911430:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488529408:192.168.202.125@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788572565.911458:0:42451:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.125@tcp lock: ffff94d412166600/0x1e260726fbbc0b62 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0aa9 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572565.911475:0:42451:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.125@tcp lock: ffff94d412166600/0x1e260726fbbc0b62 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xec901abfeb4e0aa9 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572565.911551:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488529536:192.168.202.125@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788572565.911630:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408c8bcc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572565.967112:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488529536:192.168.202.125@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788572565.967518:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488529664:192.168.202.125@tcp:503:ll_cfg_requeue.0 00000100:00100000:1.0:1788572565.967628:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408c8bcc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572565.976130:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488529664:192.168.202.125@tcp:503:ll_cfg_requeue.0 00000040:00080000:1.0:1788572565.980196:0:69760:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 64768 count 51 00000020:01000000:0.0:1788572565.980516:0:42451:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 52-51 (rc=0) 10000000:01000000:0.0:1788572565.980534:0:42451:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.202.125@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:0.0:1788572565.980546:0:42451:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412166600/0x1e260726fbbc0b62 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0aa9 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572565.980559:0:42451:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.125@tcp lock: ffff94d412166600/0x1e260726fbbc0b62 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0aa9 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788572565.980569:0:42451:0:(mgc_request.c:611:do_requeue()) updating log params 10000000:01000000:0.0:1788572565.980572:0:42451:0:(mgc_request.c:1801:mgc_process_log()) Process log params-ffff94d405cc3800 from 7 10000000:01000000:0.0:1788572565.980574:0:42451:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:0.0:1788572565.980626:0:42451:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412167000/0x1e260726fbbc0b69 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572565.980640:0:42451:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.125@tcp lock: ffff94d412167000/0x1e260726fbbc0b69 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572565.980652:0:42451:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.125@tcp lock: ffff94d412167000/0x1e260726fbbc0b69 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572565.980684:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488529792:192.168.202.125@tcp:101:ll_cfg_requeue.0 00000100:00100000:0.0:1788572565.980778:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416e1fe40 going to sleep for 11 seconds 00000100:00100000:1.0:1788572565.988694:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488529792:192.168.202.125@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788572565.988732:0:42451:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.125@tcp lock: ffff94d412167000/0x1e260726fbbc0b69 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0ab0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572565.988746:0:42451:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.125@tcp lock: ffff94d412167000/0x1e260726fbbc0b69 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xec901abfeb4e0ab0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572565.988836:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488529920:192.168.202.125@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788572565.988933:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408c8b840 going to sleep for 11 seconds 00000100:00100000:1.0:1788572565.998422:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488529920:192.168.202.125@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788572565.998488:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488530048:192.168.202.125@tcp:503:ll_cfg_requeue.0 00000100:00100000:1.0:1788572565.998590:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408c8b840 going to sleep for 11 seconds 00000100:00100000:1.0:1788572566.001378:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488530048:192.168.202.125@tcp:503:ll_cfg_requeue.0 00000040:00080000:1.0:1788572566.004571:0:69761:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 64768 count 7 00000020:01000000:0.0:1788572566.008412:0:42451:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 7-6 (rc=0) 10000000:01000000:0.0:1788572566.008452:0:42451:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.202.125@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:0.0:1788572566.008465:0:42451:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412167000/0x1e260726fbbc0b69 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0ab0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572566.008478:0:42451:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.125@tcp lock: ffff94d412167000/0x1e260726fbbc0b69 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0ab0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788572566.008489:0:42451:0:(mgc_request.c:611:do_requeue()) updating log lustre-sptlrpc 10000000:01000000:0.0:1788572566.008492:0:42451:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-sptlrpc-ffff94d4113f0000 from 1 10000000:01000000:0.0:1788572566.008495:0:42451:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:0.0:1788572566.008538:0:42451:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412167a00/0x1e260726fbbc0b70 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572566.008547:0:42451:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.125@tcp lock: ffff94d412167a00/0x1e260726fbbc0b70 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572566.008558:0:42451:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.125@tcp lock: ffff94d412167a00/0x1e260726fbbc0b70 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572566.008591:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488530176:192.168.202.125@tcp:101:ll_cfg_requeue.0 00000100:00100000:0.0:1788572566.008695:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416e1ff00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572566.023243:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488530176:192.168.202.125@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788572566.023274:0:42451:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.125@tcp lock: ffff94d412167a00/0x1e260726fbbc0b70 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0ab7 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572566.023439:0:42451:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.125@tcp lock: ffff94d412167a00/0x1e260726fbbc0b70 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xec901abfeb4e0ab7 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572566.023531:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488530304:192.168.202.125@tcp:501:ll_cfg_requeue.0 00000100:00100000:0.0:1788572566.023656:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d416e1ff00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572566.026713:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488530304:192.168.202.125@tcp:501:ll_cfg_requeue.0 10000000:01000000:0.0:1788572566.026777:0:42451:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.202.125@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:0.0:1788572566.026788:0:42451:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412167a00/0x1e260726fbbc0b70 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0ab7 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572566.026798:0:42451:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.125@tcp lock: ffff94d412167a00/0x1e260726fbbc0b70 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0ab7 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572566.149563:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572567.173375:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572568.197379:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1788572568.265978:0:51031:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572568.266006:0:51031:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:0.0:1788572568.266030:0:51031:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:0.0:1788572568.266114:0:51031:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1788572569.138556:0:2397:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.125@tcp (source: 12345-192.168.202.125@tcp) 00000100:00100000:2.0:1788572569.138827:0:42440:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875450511531520 00000100:00100000:2.0:1788572569.139149:0:42440:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875450511531520 00000100:00100000:2.0:1788572569.139254:0:42440:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.125@tcp, seq: 95 00000100:00100000:2.0:1788572569.139265:0:42440:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff94d408c56680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:e60431de-58b9-4f69-9048-11a3d82ddcdd+3:90975:x1875450511531520:12345-192.168.202.125@tcp:104: 00010000:00010000:2.0:1788572569.139283:0:42440:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: MGC192.168.202.125@tcp lock: ffff94d412167200/0x1e260726fbbc0b5b lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0xec901abfeb4e0aa2 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572569.139570:0:42440:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788572569.139579:0:42440:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff94d408c56680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:e60431de-58b9-4f69-9048-11a3d82ddcdd+3:90975:x1875450511531520:12345-192.168.202.125@tcp:104: Request processed in 316us (1037us total) trans 0 rc 0/0 00000100:00100000:2.0:1788572569.139588:0:42440:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.125@tcp, seq: 95 00010000:00010000:0.0:1788572569.148483:0:42448:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788572569.148499:0:42448:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.125@tcp lock: ffff94d412167200/0x1e260726fbbc0b5b lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0xec901abfeb4e0aa2 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572569.148518:0:42448:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412167200 already unused, calling callback (ffffffffc115cc20) 10000000:00010000:0.0:1788572569.148523:0:42448:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.125@tcp lock: ffff94d412167200/0x1e260726fbbc0b5b lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400400000000 nid: local remote: 0xec901abfeb4e0aa2 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572569.148534:0:42448:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.125@tcp lock: ffff94d412167200/0x1e260726fbbc0b5b lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1408400000000 nid: local remote: 0xec901abfeb4e0aa2 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788572569.148541:0:42448:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.125@tcp lock: ffff94d412167200/0x1e260726fbbc0b5b lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1409400000000 nid: local remote: 0xec901abfeb4e0aa2 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788572569.148547:0:42448:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788572569.148612:0:42448:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.125@tcp lock: ffff94d412167200/0x1e260726fbbc0b5b lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0xec901abfeb4e0aa2 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572569.148620:0:42448:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572569.148649:0:42448:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.125@tcp lock: ffff94d412167200/0x1e260726fbbc0b5b lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0xec901abfeb4e0aa2 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572569.148656:0:42448:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.125@tcp lock: ffff94d412167200/0x1e260726fbbc0b5b lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0xec901abfeb4e0aa2 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572569.148705:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788572569.148728:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488530432:192.168.202.125@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1788572569.148782:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572569.148785:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572569.148787:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572569.154560:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488530432:192.168.202.125@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788572569.221666:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572569.797379:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572569.797388:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572569.797392:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572569.797403:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to NID 192.168.202.125@tcp last attempt 3718 00000100:00080000:1.0:1788572569.797414:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572569.797422:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d405cc3800: import ffff94d405bab000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572569.797434:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572569.797455:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488530688/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572569.797519:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488530816:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572569.797611:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488531072:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572569.797644:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:1.0:1788572569.797650:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488530560:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572569.797693:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488530944:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572569.798230:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488530688:192.168.202.125@tcp:38:kworker.0 00010000:00080000:2.0:1788572569.812812:0:2402:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=0) req@ffff94d4049bfb80 x1875450488530688/t0(0) o38->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 520/416 e 0 to 0 dl 1788572594 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572569.812836:0:2402:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff94d405cc3800: connect to target with instance 26 00000100:00080000:2.0:1788572569.812856:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:2.0:1788572569.812862:0:2402:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788572569.812865:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 107374182407 gen 4 00000100:00100000:2.0:1788572569.812871:0:2402:0:(client.c:3156:ptlrpc_free_committed()) @@@ stopping search req@ffff94d408d3dc00 x1875450488519296/t107374182408(107374182408) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572543 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572569.812880:0:2402:0:(recover.c:53:ptlrpc_replay_next()) import ffff94d405bab000 from lustre-MDT0000_UUID committed 107374182407 last 0 00000100:00080000:2.0:1788572569.812887:0:2402:0:(client.c:3540:ptlrpc_replay_req()) @@@ REPLAY req@ffff94d408d3dc00 x1875450488519296/t107374182408(107374182408) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572543 ref 1 fl New:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572569.812900:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488530688:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788572569.812937:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488519296:192.168.202.125@tcp:36:mcreate.0 00000100:00100000:1.0:1788572569.819679:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488530560:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572569.819711:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488530944:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572569.819725:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488530816:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572569.819771:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488531072:192.168.202.125@tcp:400:kworker.0 00010000:00010000:0.0:1788572570.246705:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788572570.373654:0:42451:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788572570.373665:0:42451:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-cliir-ffff94d405cc3800 from 113 10000000:01000000:1.0:1788572570.373670:0:42451:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788572570.373729:0:42451:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412582200/0x1e260726fbbc0b77 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572570.373741:0:42451:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.125@tcp lock: ffff94d412582200/0x1e260726fbbc0b77 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572570.373753:0:42451:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.125@tcp lock: ffff94d412582200/0x1e260726fbbc0b77 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572570.373783:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488531200:192.168.202.125@tcp:101:ll_cfg_requeue.0 00000100:00100000:1.0:1788572570.373872:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408c8b840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572570.392965:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488531200:192.168.202.125@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788572570.393002:0:42451:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.125@tcp lock: ffff94d412582200/0x1e260726fbbc0b77 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0bcf expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572570.393015:0:42451:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.125@tcp lock: ffff94d412582200/0x1e260726fbbc0b77 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xec901abfeb4e0bcf expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572570.393111:0:42451:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488531328:192.168.202.125@tcp:256:ll_cfg_requeue.0 00000100:00100000:2.0:1788572570.393193:0:42451:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d407e13240 going to sleep for 11 seconds 00000100:00100000:1.0:1788572570.407767:0:42451:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:e60431de-58b9-4f69-9048-11a3d82ddcdd:42451:1875450488531328:192.168.202.125@tcp:256:ll_cfg_requeue.0 10000000:01000000:1.0:1788572570.407859:0:42451:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.202.125@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:1.0:1788572570.407869:0:42451:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412582200/0x1e260726fbbc0b77 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0bcf expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572570.407882:0:42451:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.125@tcp lock: ffff94d412582200/0x1e260726fbbc0b77 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0bcf expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572571.269434:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572572.293707:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572573.317386:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572574.341442:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572574.832306:0:2402:0:(client.c:3425:ptlrpc_replay_interpret()) @@@ got reply req@ffff94d408d3dc00 x1875450488519296/t107374182408(107374182408) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572585 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572574.832323:0:2402:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:0.0:1788572574.832326:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 107374182407 gen 4 00000100:00100000:0.0:1788572574.832332:0:2402:0:(client.c:3156:ptlrpc_free_committed()) @@@ stopping search req@ffff94d408d3dc00 x1875450488519296/t107374182408(107374182408) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572585 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572574.832340:0:2402:0:(recover.c:53:ptlrpc_replay_next()) import ffff94d405bab000 from lustre-MDT0000_UUID committed 107374182407 last 107374182408 00000100:00080000:0.0:1788572574.832347:0:2402:0:(client.c:3540:ptlrpc_replay_req()) @@@ REPLAY req@ffff94d408d3f800 x1875450488519808/t107374182409(107374182409) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572543 ref 1 fl New:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572574.832358:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488519296:192.168.202.125@tcp:36:mcreate.0 00000100:00100000:0.0:1788572574.832380:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488519808:192.168.202.125@tcp:36:rm.0 00000100:00080000:0.0:1788572574.837331:0:2402:0:(client.c:3425:ptlrpc_replay_interpret()) @@@ got reply req@ffff94d408d3f800 x1875450488519808/t107374182409(107374182409) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572590 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572574.837347:0:2402:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:0.0:1788572574.837350:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 107374182407 gen 4 00000100:00100000:0.0:1788572574.837355:0:2402:0:(client.c:3156:ptlrpc_free_committed()) @@@ stopping search req@ffff94d408d3dc00 x1875450488519296/t107374182408(107374182408) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572585 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572574.837363:0:2402:0:(recover.c:53:ptlrpc_replay_next()) import ffff94d405bab000 from lustre-MDT0000_UUID committed 107374182407 last 107374182409 00000100:00080000:0.0:1788572574.837370:0:2402:0:(client.c:3540:ptlrpc_replay_req()) @@@ REPLAY req@ffff94d408d3c380 x1875450488521728/t107374182412(107374182412) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 488/456 e 0 to 0 dl 1788572551 ref 1 fl New:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572574.837382:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488519808:192.168.202.125@tcp:36:rm.0 00000100:00100000:0.0:1788572574.837411:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488521728:192.168.202.125@tcp:36:dd.0 00000100:00080000:0.0:1788572574.841394:0:2402:0:(client.c:3425:ptlrpc_replay_interpret()) @@@ got reply req@ffff94d408d3c380 x1875450488521728/t107374182412(107374182412) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 488/456 e 0 to 0 dl 1788572590 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572574.841419:0:2402:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:0.0:1788572574.841421:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 107374182407 gen 4 00000100:00100000:0.0:1788572574.841427:0:2402:0:(client.c:3156:ptlrpc_free_committed()) @@@ stopping search req@ffff94d408d3dc00 x1875450488519296/t107374182408(107374182408) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572585 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572574.841435:0:2402:0:(recover.c:53:ptlrpc_replay_next()) import ffff94d405bab000 from lustre-MDT0000_UUID committed 107374182407 last 107374182412 00000100:00080000:0.0:1788572574.841442:0:2402:0:(client.c:3540:ptlrpc_replay_req()) @@@ REPLAY req@ffff94d408c54a80 x1875450488522112/t107374182413(107374182413) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 640/608 e 0 to 0 dl 1788572551 ref 1 fl New:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788572574.841454:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488521728:192.168.202.125@tcp:36:dd.0 00000100:00100000:0.0:1788572574.841476:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488522112:192.168.202.125@tcp:101:dd.0 00000100:00080000:0.0:1788572574.848194:0:2402:0:(client.c:3425:ptlrpc_replay_interpret()) @@@ got reply req@ffff94d408c54a80 x1875450488522112/t107374182413(107374182413) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 640/608 e 0 to 0 dl 1788572590 ref 2 fl Interpret:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:0.0:1788572574.848237:0:2402:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:0.0:1788572574.848240:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 107374182407 gen 4 00000100:00100000:0.0:1788572574.848246:0:2402:0:(client.c:3156:ptlrpc_free_committed()) @@@ stopping search req@ffff94d408d3dc00 x1875450488519296/t107374182408(107374182408) o36->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572585 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572574.848256:0:2402:0:(recover.c:53:ptlrpc_replay_next()) import ffff94d405bab000 from lustre-MDT0000_UUID committed 107374182407 last 107374182413 00000100:00080000:0.0:1788572574.848260:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:2.0:1788572574.848719:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488522112:192.168.202.125@tcp:101:dd.0 00010000:00080000:0.0:1788572574.848998:0:69808:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff94d405cc3800 to lustre-MDT0000_UUID@192.168.202.125@tcp 00010000:00010000:0.0:1788572574.849003:0:69808:0:(ldlm_request.c:2888:ldlm_cancel_unused_locks_for_replay()) Dropping as many unused locks as possible before replay for namespace lustre-MDT0000-mdc-ffff94d405cc3800 (7) 00000080:00200000:0.0:1788572574.849048:0:69808:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1788572574.849063:0:69808:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:0.0:1788572574.849130:0:69808:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167400/0x1e260726fbbc0b46 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0xec901abfeb4e0a16 expref: -99 pid: 69531 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1788572574.849165:0:69808:0:(vvp_object.c:117:vvp_conf_set()) [0x20000a811:0x9:0x0]: losing layout lock 00010000:00010000:0.0:1788572574.849323:0:69808:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031600/0x1e260726fbbc0b54 lrc: 0/0,0 mode: --/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0xec901abfeb4e0a5c expref: -99 pid: 69533 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788572574.849348:0:69808:0:(vvp_object.c:117:vvp_conf_set()) [0x20000a811:0x9:0x0]: losing layout lock 00010000:00010000:0.0:1788572574.849362:0:69808:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d400031e00/0x1e260726fbbc0b2a lrc: 0/0,0 mode: --/CR res: [0x20000a811:0x9:0x0].0x0 bits 0xa/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0xec901abfeb4e09f3 expref: -99 pid: 69530 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572574.849382:0:69808:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b1c lrc: 0/0,0 mode: --/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0xec901abfeb4e09d7 expref: -99 pid: 69530 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572574.849412:0:69808:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1788572574.849468:0:69808:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:0.0:1788572574.849488:0:69808:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167c00/0x1e260726fbbc0add lrc: 0/0,0 mode: --/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x40/0x0 rrc: 4 type: IBT gid 0 flags: 0x804829400000000 nid: local remote: 0xec901abfeb4e08e9 expref: -99 pid: 68711 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788572574.849509:0:69808:0:(vvp_object.c:117:vvp_conf_set()) [0x20000a811:0x6:0x0]: losing layout lock 00010000:00010000:0.0:1788572574.849520:0:69808:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412582e00/0x1e260726fbbc0ad6 lrc: 0/0,0 mode: --/CR res: [0x20000a811:0x6:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0xec901abfeb4e08e2 expref: -99 pid: 68711 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572574.849532:0:69808:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412583400/0x1e260726fbbc0acf lrc: 0/0,0 mode: --/PR res: [0x20000a811:0x6:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0xec901abfeb4e08db expref: -99 pid: 68711 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572574.849554:0:69808:0:(ldlm_request.c:2901:ldlm_cancel_unused_locks_for_replay()) Canceled 0 unused locks from namespace lustre-MDT0000-mdc-ffff94d405cc3800 00000100:00080000:0.0:1788572574.850107:0:69808:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to REPLAY_WAIT 00000100:00100000:3.0:1788572574.850576:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488531456:192.168.202.125@tcp:400:ldlm_lock_repla.0 00000100:00100000:3.0:1788572574.864751:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 107374182413 gen 4 00000100:00080000:3.0:1788572574.864808:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from REPLAY_WAIT to RECOVER 00000100:00080000:3.0:1788572574.864830:0:2402:0:(client.c:3234:ptlrpc_resend_req()) @@@ going to resend req@ffff94d408d3f100 x1875450488531456/t0(0) o400->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572590 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572574.864841:0:2402:0:(client.c:3243:ptlrpc_resend_req()) @@@ it has reply, so skip it req@ffff94d408d3f100 x1875450488531456/t0(0) o400->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572590 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572574.864849:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from RECOVER to FULL 00000100:02000000:2.0:1788572574.865334:0:2402:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) lustre-MDT0000-mdc-ffff94d405cc3800: Connection restored to 192.168.202.125@tcp (at 192.168.202.125@tcp) 00000100:00100000:2.0:1788572574.865343:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488531456:192.168.202.125@tcp:400:ldlm_lock_repla.0 00000100:00100000:3.0:1788572574.920042:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488531584:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572574.920257:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488531840:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572574.920292:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788572574.920298:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488531712:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572574.920318:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488531968:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572574.943663:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488531584:192.168.202.125@tcp:400:kworker.0 00010000:00010000:3.0:1788572574.944108:0:2406:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 0/0 locks to regular blp list, 1 blwis in pool 00000100:00100000:3.0:1788572574.944127:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488531840:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572574.944159:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488531712:192.168.202.125@tcp:400:kworker.0 00010000:00010000:2.0:1788572574.944266:0:42449:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (1 blwis in pool) 00010000:00010000:2.0:1788572574.944284:0:42449:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572574.944289:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572574.944290:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572574.944292:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572574.944407:0:2406:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 0/0 locks to regular blp list, 0 blwis in pool 00000100:00100000:3.0:1788572574.944415:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488531968:192.168.202.125@tcp:400:kworker.0 00010000:00010000:0.0:1788572574.944665:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572575.365396:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572576.398916:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572577.414339:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572578.439564:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:3.0:1788572579.438170:0:69862:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572579.438184:0:69862:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:3.0:1788572579.438685:0:69862:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572579.441771:0:69862:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 69862 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572579.441789:0:69862:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 69862 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572579.441799:0:69862:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 69862 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572579.441826:0:69862:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69862:1875450488532096:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572579.441951:0:69862:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d407e13d80 going to sleep for 11 seconds 00010000:00010000:2.0:1788572579.461506:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572579.467681:0:69862:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69862:1875450488532096:192.168.202.125@tcp:101:lfs.0 00010000:00010000:3.0:1788572579.467709:0:69862:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572579.467726:0:69862:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572579.467737:0:69862:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572579.467746:0:69862:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572579.467758:0:69862:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572579.467768:0:69862:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572579.467774:0:69862:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572579.467785:0:69862:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408c60a80 x1875450488532096/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572595 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572579.467800:0:69862:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572579.467814:0:69862:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788572579.467821:0:69862:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572579.467850:0:69862:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:3.0:1788572579.467857:0:69862:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0b7e 00000080:00010000:3.0:1788572579.467861:0:69862:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:3.0:1788572579.467864:0:69862:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b7e from it ffffb68a422bfcb8 00010000:00010000:3.0:1788572579.467867:0:69862:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572579.467874:0:69862:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788572579.473868:0:69862:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788572579.473879:0:69862:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572579.473902:0:69862:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572579.473920:0:69862:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572579.474481:0:69862:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572579.474494:0:69862:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572579.474503:0:69862:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572579.474506:0:69862:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0b7e 00000080:00010000:2.0:1788572579.474515:0:69862:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b7e from it ffffb68a422bfcb8 00010000:00010000:2.0:1788572579.474519:0:69862:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572579.474528:0:69862:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572579.479345:0:69862:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572579.479387:0:69862:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572579.479405:0:69862:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572579.479422:0:69862:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572579.479431:0:69862:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572579.479438:0:69862:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572579.479441:0:69862:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0b7e 00000080:00010000:2.0:1788572579.479449:0:69862:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b7e from it ffffb68a422bfbb0 00010000:00010000:2.0:1788572579.479454:0:69862:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572579.479463:0:69862:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572579.479478:0:69862:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 44 00000080:00200008:2.0:1788572579.479488:0:69862:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 100000 00000080:00200080:2.0:1788572579.479770:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:2.0:1788572579.479788:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d41541e800 uarg=0000000000000000 00800000:00000004:2.0:1788572579.479795:0:69862:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0000-mdc-ffff94d405cc3800: age 3680, max_age 3731 00000100:00100000:2.0:1788572579.479840:0:69862:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69862:1875450488532224:192.168.202.125@tcp:41:lfs.0 00000100:00100000:2.0:1788572579.480510:0:69862:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d407e13d80 going to sleep for 11 seconds 00000100:00100000:0.0:1788572579.488142:0:69862:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69862:1875450488532224:192.168.202.125@tcp:41:lfs.0 00800000:00000004:0.0:1788572579.488170:0:69862:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0000-mdc-ffff94d405cc3800: update ffff94d412909e10 cache blocks 317130/353529 objects 1023602/1024000 00000080:00200080:0.0:1788572579.488792:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:0.0:1788572579.488814:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084ddc00 uarg=0000000000000000 00800000:00000004:0.0:1788572579.488822:0:69862:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0001-mdc-ffff94d405cc3800: age 3680, max_age 3731 00000100:00100000:0.0:1788572579.488860:0:69862:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69862:1875450488532352:192.168.202.125@tcp:41:lfs.0 00000100:00100000:0.0:1788572579.488943:0:69862:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d412842900 going to sleep for 11 seconds 00000100:00100000:3.0:1788572579.497195:0:69862:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69862:1875450488532352:192.168.202.125@tcp:41:lfs.0 00800000:00000004:3.0:1788572579.497278:0:69862:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0001-mdc-ffff94d405cc3800: update ffff94d407db9e10 cache blocks 317207/353529 objects 1023684/1024000 00000080:00200080:3.0:1788572579.497368:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497391:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497400:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497405:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497410:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497413:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497417:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497420:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497424:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497427:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497432:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497435:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497439:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497444:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497448:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497453:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497457:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497460:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497464:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497468:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497473:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497477:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497481:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497484:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497488:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497491:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497495:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497498:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497503:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497506:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497510:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497514:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497518:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497521:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497525:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497529:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497533:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497537:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497541:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497544:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497549:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497553:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497557:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497560:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497564:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497567:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497571:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497574:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497578:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497581:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497586:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497589:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497593:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497597:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497729:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497737:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497745:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497750:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497755:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497759:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497763:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497767:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497771:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497774:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497778:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497782:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497785:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497789:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497793:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497796:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497800:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497804:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497808:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497812:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497816:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497819:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497823:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497826:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497830:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497833:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497837:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497840:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497843:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497847:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497850:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497852:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497855:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497858:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497861:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497864:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497868:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497871:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497874:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497877:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497880:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497883:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497886:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497889:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497893:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497896:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497900:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497903:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497906:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497909:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497912:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497915:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497919:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497921:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497925:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497928:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497932:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497935:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497939:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497942:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497946:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497949:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497952:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497956:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497959:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497962:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.497965:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.497969:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.498042:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.498051:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.498057:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00800000:00000080:3.0:1788572579.498061:0:69862:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.498065:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.498070:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00020000:00000004:3.0:1788572579.498077:0:69862:0:(obd_class.h:1091:obd_statfs()) lustre-OST0000-osc-ffff94d405cc3800: age 3680, max_age 3731 00000100:00100000:3.0:1788572579.498118:0:69862:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69862:1875450488532480:192.168.202.125@tcp:13:lfs.0 00000100:00100000:3.0:1788572579.498264:0:69862:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405411cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572579.506061:0:69862:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69862:1875450488532480:192.168.202.125@tcp:13:lfs.0 00020000:00000004:3.0:1788572579.506095:0:69862:0:(obd_class.h:1117:obd_statfs()) lustre-OST0000-osc-ffff94d405cc3800: update ffff94d407dbd190 cache blocks 901369/958279 objects 261731/262144 00000080:00200080:3.0:1788572579.506167:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.506186:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00020000:00000004:3.0:1788572579.506194:0:69862:0:(obd_class.h:1091:obd_statfs()) lustre-OST0001-osc-ffff94d405cc3800: age 3680, max_age 3731 00000100:00100000:3.0:1788572579.506559:0:69862:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69862:1875450488532608:192.168.202.125@tcp:13:lfs.0 00000100:00100000:3.0:1788572579.506739:0:69862:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405411cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572579.523282:0:69862:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:69862:1875450488532608:192.168.202.125@tcp:13:lfs.0 00020000:00000004:3.0:1788572579.523311:0:69862:0:(obd_class.h:1117:obd_statfs()) lustre-OST0001-osc-ffff94d405cc3800: update ffff94d407dbb7d0 cache blocks 901113/958279 objects 261732/262144 00000080:00200080:3.0:1788572579.523392:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.523409:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.523417:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.523421:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.523424:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.523428:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.523431:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.523435:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.523439:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.523442:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.523446:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.523450:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.523454:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.523457:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.523461:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.523464:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.523468:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.523471:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.523474:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.523477:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.523481:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.523483:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.523487:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.523490:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.523494:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.523498:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.523502:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.523505:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.523509:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.523513:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.523516:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.523519:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.523523:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.523526:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.523531:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.523534:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.523538:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.523541:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.523544:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.523547:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.523551:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.523554:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.523558:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.523562:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.523565:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.523568:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.523573:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.523576:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.523579:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.523583:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.523587:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.523590:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.523593:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.523597:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.523950:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524094:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524105:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524110:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524115:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524119:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524123:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524127:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524130:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524134:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524138:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524141:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524144:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524147:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524151:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524154:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524158:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524161:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524165:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524168:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524173:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524177:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524181:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524184:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524188:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524192:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524195:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524199:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524336:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524344:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524349:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524361:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524367:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524372:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524376:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524379:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524387:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524392:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524395:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524399:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524403:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524406:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524410:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524414:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524418:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524422:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524426:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524429:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524433:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524437:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524441:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524444:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524447:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524450:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524453:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524456:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524459:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524463:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524466:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524469:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524473:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524476:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524479:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524482:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524485:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524488:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524492:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524495:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524498:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524501:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200080:3.0:1788572579.524506:0:69862:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7ffcdc9e7e80 00020000:00000080:3.0:1788572579.524509:0:69862:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d38c00 uarg=0000000000000000 00000080:00200008:3.0:1788572579.524523:0:69862:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 100000 00000100:00100000:1.0:1788572580.037523:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488532736:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572580.037667:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488532992:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572580.037695:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488533248:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572580.037722:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:1.0:1788572580.037728:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488532864:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572580.037750:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488533120:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572580.050411:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488533120:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572580.050430:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488532992:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572580.050441:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488533248:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572580.050471:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488532864:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572580.054199:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e60431de-58b9-4f69-9048-11a3d82ddcdd:2403:1875450488532736:192.168.202.125@tcp:400:kworker.0 00010000:00010000:0.0:1788572580.485380:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572581.509372:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572582.533360:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572583.573408:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572584.581343:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572585.159346:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488533376:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572585.159451:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488533632:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572585.159481:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488533888:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572585.159511:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788572585.159518:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488533504:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572585.159541:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488533760:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572585.163013:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488533760:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572585.163040:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488533888:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572585.164273:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488533504:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572585.164290:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488533632:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572585.167290:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488533376:192.168.202.125@tcp:400:kworker.0 00010000:00010000:3.0:1788572585.608128:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788572586.281707:0:70143:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 00010000:00010000:2.0:1788572586.630887:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572587.655200:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788572588.142437:0:70291:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 00010000:00010000:3.0:1788572588.678376:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572589.703494:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:0.0:1788572590.116012:0:70329:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788572590.116028:0:70329:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:0.0:1788572590.116054:0:70329:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572590.116072:0:70329:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572590.116087:0:70329:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788572590.116096:0:70329:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572590.116104:0:70329:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572590.116108:0:70329:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0b7e 00000080:00010000:0.0:1788572590.116116:0:70329:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b7e from it ffffb68a41de7cb8 00010000:00010000:0.0:1788572590.116121:0:70329:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572590.116130:0:70329:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:0.0:1788572590.117060:0:70329:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1788572590.117068:0:70329:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:0.0:1788572590.117081:0:70329:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572590.117091:0:70329:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788572590.117101:0:70329:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788572590.117108:0:70329:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572590.117114:0:70329:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572590.117117:0:70329:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0b7e 00000080:00010000:0.0:1788572590.117121:0:70329:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b7e from it ffffb68a41de7cb8 00010000:00010000:0.0:1788572590.117124:0:70329:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788572590.117131:0:70329:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1788572590.117328:0:70329:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:0.0:1788572590.117337:0:70329:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572590.117344:0:70329:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1788572590.117354:0:70329:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1788572590.117360:0:70329:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572590.117364:0:70329:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572590.117367:0:70329:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0b7e 00000080:00010000:0.0:1788572590.117370:0:70329:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b7e from it ffffb68a41de7bb0 00010000:00010000:0.0:1788572590.117374:0:70329:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1788572590.117381:0:70329:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:0.0:1788572590.117389:0:70329:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 44 00000080:00200008:0.0:1788572590.117395:0:70329:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 100000 00000080:00200080:0.0:1788572590.117546:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:0.0:1788572590.117561:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00800000:00000004:0.0:1788572590.117568:0:70329:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0000-mdc-ffff94d405cc3800: age 3732, max_age 3742 00000100:00100000:0.0:1788572590.117611:0:70329:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70329:1875450488534016:192.168.202.125@tcp:41:lfs.0 00000100:00100000:0.0:1788572590.117691:0:70329:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408619300 going to sleep for 11 seconds 00000100:00100000:0.0:1788572590.133747:0:70329:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70329:1875450488534016:192.168.202.125@tcp:41:lfs.0 00800000:00000004:0.0:1788572590.133776:0:70329:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0000-mdc-ffff94d405cc3800: update ffff94d412909e10 cache blocks 317130/353529 objects 1023602/1024000 00000080:00200080:0.0:1788572590.133855:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:0.0:1788572590.133872:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084dc400 uarg=0000000000000000 00800000:00000004:0.0:1788572590.133882:0:70329:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0001-mdc-ffff94d405cc3800: age 3732, max_age 3742 00000100:00100000:0.0:1788572590.133916:0:70329:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70329:1875450488534144:192.168.202.125@tcp:41:lfs.0 00000100:00100000:0.0:1788572590.134019:0:70329:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408619300 going to sleep for 11 seconds 00000100:00100000:1.0:1788572590.139737:0:70329:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70329:1875450488534144:192.168.202.125@tcp:41:lfs.0 00800000:00000004:1.0:1788572590.139773:0:70329:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0001-mdc-ffff94d405cc3800: update ffff94d407db9e10 cache blocks 317207/353529 objects 1023684/1024000 00000080:00200080:1.0:1788572590.139860:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.139879:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.139887:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.139891:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.139896:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.139900:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.139904:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.139908:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.139912:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.139915:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.139920:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.139923:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.139927:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.139930:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.139934:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.139937:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.139951:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.139957:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.139961:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.139964:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.139968:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.139972:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.139976:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.139979:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.139983:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.139987:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.139991:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.139994:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.139998:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140001:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140004:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140008:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140011:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140014:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140018:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140022:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140026:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140029:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140033:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140037:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140041:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140044:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140048:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140051:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140055:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140058:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140062:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140065:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140068:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140072:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140075:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140079:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140082:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140085:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140090:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140093:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140097:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140101:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140104:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140108:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140111:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140115:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140119:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140122:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140126:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140129:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140133:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140136:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140140:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140143:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140147:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140150:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140154:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140157:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140161:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140165:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140169:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140173:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140177:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140181:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140185:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140188:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140192:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140196:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140199:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140256:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140262:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140266:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140271:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140274:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140278:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140282:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140286:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140289:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140293:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140296:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140299:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140302:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140306:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140308:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140311:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140315:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140318:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140321:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140325:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140328:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140332:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140335:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140338:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140341:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140344:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140347:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140351:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140353:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140357:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140360:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140363:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140366:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140370:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140373:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140377:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140380:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140384:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140387:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140390:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00800000:00000080:1.0:1788572590.140393:0:70329:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00000080:00200080:1.0:1788572590.140397:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:1.0:1788572590.140404:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00020000:00000004:1.0:1788572590.140413:0:70329:0:(obd_class.h:1091:obd_statfs()) lustre-OST0000-osc-ffff94d405cc3800: age 3732, max_age 3742 00000100:00100000:1.0:1788572590.140456:0:70329:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70329:1875450488534272:192.168.202.125@tcp:13:lfs.0 00000100:00100000:1.0:1788572590.140540:0:70329:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411e03540 going to sleep for 11 seconds 00000100:00100000:1.0:1788572590.145135:0:70329:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70329:1875450488534272:192.168.202.125@tcp:13:lfs.0 00020000:00000004:1.0:1788572590.145167:0:70329:0:(obd_class.h:1117:obd_statfs()) lustre-OST0000-osc-ffff94d405cc3800: update ffff94d407dbd190 cache blocks 901369/958279 objects 261731/262144 00000080:00200080:1.0:1788572590.145273:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:1.0:1788572590.145291:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d408d55800 uarg=0000000000000000 00020000:00000004:1.0:1788572590.145299:0:70329:0:(obd_class.h:1091:obd_statfs()) lustre-OST0001-osc-ffff94d405cc3800: age 3732, max_age 3742 00000100:00100000:1.0:1788572590.145332:0:70329:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70329:1875450488534400:192.168.202.125@tcp:13:lfs.0 00000100:00100000:1.0:1788572590.145406:0:70329:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411e03540 going to sleep for 11 seconds 00000100:00100000:0.0:1788572590.149488:0:70329:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70329:1875450488534400:192.168.202.125@tcp:13:lfs.0 00020000:00000004:0.0:1788572590.149522:0:70329:0:(obd_class.h:1117:obd_statfs()) lustre-OST0001-osc-ffff94d405cc3800: update ffff94d407dbb7d0 cache blocks 901113/958279 objects 261732/262144 00000080:00200080:0.0:1788572590.149599:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149616:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149624:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149629:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149633:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149637:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149641:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149645:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149649:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149652:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149656:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149660:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149664:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149667:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149671:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149675:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149679:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149683:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149687:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149691:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149697:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149701:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149706:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149710:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149714:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149717:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149721:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149725:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149729:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149733:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149737:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149740:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149744:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149747:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149751:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149754:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149758:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149761:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149765:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149769:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149773:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149777:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149781:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149784:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149789:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149793:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149797:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149801:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149805:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149808:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149812:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149816:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149820:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149824:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149828:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149832:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149836:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149839:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149843:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149847:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149852:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149855:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149859:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149862:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149866:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149870:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149874:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149877:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149881:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149884:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149889:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149892:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149897:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149900:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149904:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149907:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149911:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149914:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149918:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149921:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149925:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149928:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149932:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149936:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149950:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149954:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149958:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149962:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149965:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149969:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149973:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149977:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149981:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149985:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149989:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.149993:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.149997:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.150000:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.150004:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.150008:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.150011:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.150015:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.150018:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.150021:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.150025:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.150028:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.150031:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.150035:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.150038:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.150041:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.150044:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.150047:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.150050:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.150053:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.150057:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.150060:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.150063:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.150066:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.150070:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.150073:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.150076:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.150079:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.150083:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.150087:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200080:0.0:1788572590.150091:0:70329:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c00866a4 arg=7fff7c533a70 00020000:00000080:0.0:1788572590.150095:0:70329:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d405cc3800: cmd=c00866a4 len=592 karg=ffff94d4084df800 uarg=0000000000000000 00000080:00200008:0.0:1788572590.150110:0:70329:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 100000 00000080:00200000:0.0:1788572590.215529:0:70346:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:0.0:1788572590.215563:0:70346:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572590.215581:0:70346:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1788572590.215598:0:70346:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:0.0:1788572590.215606:0:70346:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572590.215614:0:70346:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572590.215618:0:70346:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0b7e 00000080:00010000:0.0:1788572590.215625:0:70346:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b7e from it ffffb68a41fc7bf8 00010000:00010000:0.0:1788572590.215629:0:70346:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1788572590.215637:0:70346:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:0.0:1788572590.215652:0:70346:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572590.215665:0:70346:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f132a.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), file ffff94d4083f7380, open_flags 18800, mode 0 opened 0 00000080:00200000:0.0:1788572590.215671:0:70346:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f132a.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), intent=open 00000002:00010000:0.0:1788572590.215685:0:70346:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f132a.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0304001 00010000:00010000:0.0:1788572590.215735:0:70346:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b85 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 70346 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572590.215744:0:70346:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b85 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 70346 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572590.215756:0:70346:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572590.215768:0:70346:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408d3e300 x1875450488534656/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 800/131992 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572590.215777:0:70346:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b85 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 70346 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572590.215798:0:70346:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70346:1875450488534656:192.168.202.125@tcp:101:lfs.0 00000100:00100000:0.0:1788572590.215990:0:70346:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408619000 going to sleep for 11 seconds 00000100:00100000:0.0:1788572590.232668:0:70346:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70346:1875450488534656:192.168.202.125@tcp:101:lfs.0 00010000:00010000:0.0:1788572590.232692:0:70346:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b85 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 70346 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572590.232707:0:70346:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b85 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 70346 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572590.232714:0:70346:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b85 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 70346 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572590.232722:0:70346:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b85 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 70346 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572590.232735:0:70346:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b85 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 70346 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572590.232743:0:70346:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412167600 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:0.0:1788572590.232749:0:70346:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b85 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 70346 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572590.232756:0:70346:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b85 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 70346 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572590.232763:0:70346:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b85 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 70346 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572590.232770:0:70346:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788572590.232776:0:70346:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b85 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 70346 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572590.232782:0:70346:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b85 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 70346 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572590.233446:0:70346:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0xb, status=-20 req@ffff94d408d3e300 x1875450488534656/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 800/608 e 0 to 0 dl 1788572606 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572590.233465:0:70346:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc -20 00000080:00200000:1.0:1788572590.234018:0:70346:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572590.234040:0:70346:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572590.234068:0:70346:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1788572590.234081:0:70346:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1788572590.234088:0:70346:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572590.234094:0:70346:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572590.234098:0:70346:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0b7e 00000080:00010000:1.0:1788572590.234104:0:70346:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b7e from it ffffb68a41fc7bb0 00010000:00010000:1.0:1788572590.234107:0:70346:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1788572590.234115:0:70346:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:1.0:1788572590.234135:0:70346:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 44 00000080:00200008:1.0:1788572590.234144:0:70346:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 304000 00000080:00200080:1.0:1788572590.234174:0:70346:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c0086915 arg=1a8d3d0 00000100:00100000:1.0:1788572590.234273:0:70346:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70346:1875450488534784:192.168.202.125@tcp:34:lfs.0 00000100:00100000:1.0:1788572590.234354:0:70346:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411e036c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572590.237638:0:70346:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70346:1875450488534784:192.168.202.125@tcp:34:lfs.0 00000080:00200008:2.0:1788572590.237831:0:70346:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 304000 00000080:00200000:2.0:1788572590.254929:0:70350:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572590.254973:0:70350:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572590.254991:0:70350:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1788572590.255007:0:70350:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1788572590.255015:0:70350:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572590.255023:0:70350:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572590.255027:0:70350:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0b7e 00000080:00010000:2.0:1788572590.255035:0:70350:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b7e from it ffffb68a40d17bf8 00010000:00010000:2.0:1788572590.255039:0:70350:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1788572590.255048:0:70350:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:2.0:1788572590.255064:0:70350:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572590.255079:0:70350:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f132a.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), file ffff94d402bfd980, open_flags 18800, mode 0 opened 0 00000080:00200000:2.0:1788572590.255086:0:70350:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f132a.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), intent=open 00000002:00010000:2.0:1788572590.255102:0:70350:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f132a.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0304001 00010000:00010000:2.0:1788572590.255154:0:70350:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0b8c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 70350 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572590.255163:0:70350:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0b8c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 70350 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572590.255176:0:70350:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572590.255189:0:70350:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c61180 x1875450488535040/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 800/131992 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572590.255198:0:70350:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0b8c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 70350 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572590.255389:0:70350:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70350:1875450488535040:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572590.255605:0:70350:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405388f00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572590.262671:0:70350:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70350:1875450488535040:192.168.202.125@tcp:101:lfs.0 00010000:00010000:2.0:1788572590.262698:0:70350:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0b8c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 70350 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572590.262712:0:70350:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0b8c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 70350 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572590.262719:0:70350:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0b8c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 70350 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572590.262727:0:70350:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0b8c lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 70350 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572590.262736:0:70350:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0b8c lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 70350 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572590.262743:0:70350:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412536c00 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:2.0:1788572590.262747:0:70350:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0b8c lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 70350 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572590.262753:0:70350:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0b8c lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 70350 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572590.262760:0:70350:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0b8c lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 70350 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572590.262767:0:70350:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572590.262773:0:70350:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0b8c lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 70350 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572590.262779:0:70350:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536c00/0x1e260726fbbc0b8c lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 70350 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572590.262804:0:70350:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0xb, status=-20 req@ffff94d408c61180 x1875450488535040/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 800/608 e 0 to 0 dl 1788572606 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572590.262819:0:70350:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc -20 00000080:00200000:2.0:1788572590.262986:0:70350:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572590.263007:0:70350:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572590.263021:0:70350:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:2.0:1788572590.263034:0:70350:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:2.0:1788572590.263041:0:70350:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572590.263046:0:70350:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572590.263050:0:70350:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0b7e 00000080:00010000:2.0:1788572590.263055:0:70350:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b7e from it ffffb68a40d17bb0 00010000:00010000:2.0:1788572590.263058:0:70350:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:2.0:1788572590.263067:0:70350:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000080:00200000:2.0:1788572590.263079:0:70350:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 44 00000080:00200008:2.0:1788572590.263088:0:70350:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 304000 00000080:00200080:2.0:1788572590.263119:0:70350:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) cmd=c0086915 arg=14b13d0 00000100:00100000:2.0:1788572590.263175:0:70350:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70350:1875450488535168:192.168.202.125@tcp:34:lfs.0 00000100:00100000:2.0:1788572590.263312:0:70350:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405388f00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572590.267869:0:70350:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70350:1875450488535168:192.168.202.125@tcp:34:lfs.0 00000080:00200008:2.0:1788572590.272797:0:70350:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), flags 304000 00000100:00100000:1.0:1788572590.292464:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488535296:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572590.298186:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488535296:192.168.202.125@tcp:400:kworker.0 00000080:00200000:0.0:1788572590.318582:0:70355:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:0.0:1788572590.318612:0:70355:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572590.318631:0:70355:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:0.0:1788572590.318647:0:70355:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000080:00200000:0.0:1788572590.318656:0:70355:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572590.318664:0:70355:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572590.318668:0:70355:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0b7e 00000080:00010000:0.0:1788572590.318676:0:70355:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b7e from it ffffb68a41c27bf8 00010000:00010000:0.0:1788572590.318680:0:70355:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:0.0:1788572590.318690:0:70355:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:0.0:1788572590.318703:0:70355:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572590.318717:0:70355:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f132a.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), file ffff94d412b82a80, open_flags 8000, mode 0 opened 0 00000080:00200000:0.0:1788572590.318723:0:70355:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f132a.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), intent=open 00000002:00010000:0.0:1788572590.318737:0:70355:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f132a.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100001 00010000:00010000:0.0:1788572590.318787:0:70355:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b93 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 70355 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572590.318796:0:70355:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b93 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 70355 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572590.318808:0:70355:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572590.318823:0:70355:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408d3e300 x1875450488535552/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 800/131992 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572590.318831:0:70355:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b93 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 70355 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572590.318854:0:70355:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job md5sum:9bf287c0-f582-4b83-84c1-15e766218620:70355:1875450488535552:192.168.202.125@tcp:101:md5sum.0 00000100:00100000:0.0:1788572590.319040:0:70355:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408619000 going to sleep for 11 seconds 00000100:00100000:1.0:1788572590.324619:0:70355:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job md5sum:9bf287c0-f582-4b83-84c1-15e766218620:70355:1875450488535552:192.168.202.125@tcp:101:md5sum.0 00010000:00010000:1.0:1788572590.324652:0:70355:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b93 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f10 expref: -99 pid: 70355 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572590.324675:0:70355:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b93 lrc: 4/1,0 mode: --/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f10 expref: -99 pid: 70355 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572590.324683:0:70355:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b93 lrc: 4/1,0 mode: --/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f10 expref: -99 pid: 70355 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572590.324693:0:70355:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b93 lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0f10 expref: -99 pid: 70355 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572590.324705:0:70355:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b93 lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f10 expref: -99 pid: 70355 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572590.324721:0:70355:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b93 lrc: 5/2,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0f10 expref: -99 pid: 70355 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572590.324730:0:70355:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b93 lrc: 5/2,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0f10 expref: -99 pid: 70355 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572590.324737:0:70355:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b93 lrc: 4/1,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0f10 expref: -99 pid: 70355 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572590.324748:0:70355:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff94d408d3e300 x1875450488535552/t111669149697(111669149697) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 800/872 e 0 to 0 dl 1788572606 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'md5sum.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572590.324761:0:70355:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d408d3e300 x1875450488535552/t111669149697(111669149697) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 800/872 e 0 to 0 dl 1788572606 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'md5sum.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572590.324773:0:70355:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 264 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b93 lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f10 expref: -99 pid: 70355 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788572590.324787:0:70355:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b93 lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f10 expref: -99 pid: 70355 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572590.324797:0:70355:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589977617/9 (0 0) 00000080:00200000:1.0:1788572590.324803:0:70355:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572590.324886:0:70355:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572590.324896:0:70355:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:1.0:1788572590.324918:0:70355:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:1.0:1788572590.325006:0:70355:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000008:00010000:1.0:1788572590.329524:0:70355:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 0 [cma]time: 0 0 0 by oinfo size 0 blocks 0 [cma]time 0 0 1788572535 ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 1/0,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00000008:00010000:1.0:1788572590.329566:0:70355:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 0 [cma]time: 0 0 0 by oinfo size 2097152 blocks 2048 [cma]time 1788572535 1788572535 1788572535 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 1/0,0 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00000008:00010000:1.0:1788572590.329581:0:70355:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 0 [cma]time: 0 0 0 by oinfo size 0 blocks 0 [cma]time 1788572535 1788572535 0 ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 1/0,0 mode: PW/PW res: [0x280000401:0x703:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a08 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788572590.329669:0:70355:0:(namei.c:147:ll_iget()) got inode: [0x20000a811:0x9:0x0](ffff94d410bc3248): rc = 0 00000080:00200000:1.0:1788572590.329711:0:70355:0:(file.c:6826:ll_layout_conf()) [0x20000a811:0x9:0x0]: layout version change: 4294967294 -> 4 00000080:00010000:1.0:1788572590.329724:0:70355:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a811:0x9:0x0](ffff94d410bc3248) for lock 0x1e260726fbbc0b93 00000080:00010000:1.0:1788572590.329738:0:70355:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a811:0x9:0x0](ffff94d410bc3248) 00000080:00200008:1.0:1788572590.329753:0:70355:0:(file.c:1090:ll_file_open()) START file f132a.replay-single:[0x20000a811:0x9:0x0](ffff94d410bc3248), flags 100000 00000002:00100000:1.0:1788572590.329790:0:70355:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d408d3e300 x1875450488535552/t111669149697(111669149697) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 800/872 e 0 to 0 dl 1788572606 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'md5sum.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788572590.329843:0:70355:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b93 from it ffff94d416ec4600 00010000:00010000:1.0:1788572590.329849:0:70355:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b93 lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4e0f10 expref: -99 pid: 70355 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572590.329860:0:70355:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b93 lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e0f10 expref: -99 pid: 70355 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572590.329886:0:70355:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x20000a811:0x9:0x0](ffff94d410bc3248), inode mode 81a4 mask 44 00000080:00200008:2.0:1788572590.336392:0:70355:0:(file.c:6285:ll_getattr_dentry()) START file f132a.replay-single:[0x20000a811:0x9:0x0](ffff94d410bc3248), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572590.336409:0:70355:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x20000a811:0x9:0x0](ffff94d410bc3248),name=f132a.replay-single 00000002:00010000:2.0:1788572590.336438:0:70355:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x20000a811:0x9:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572590.336456:0:70355:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b93 lrc: 2/1,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e0f10 expref: -99 pid: 70355 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572590.336471:0:70355:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b93 lrc: 2/1,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e0f10 expref: -99 pid: 70355 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572590.336479:0:70355:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572590.336486:0:70355:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000a811:0x9:0x0](ffff94d410bc3248) 00000080:00010000:2.0:1788572590.336489:0:70355:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000a811:0x9:0x0](ffff94d410bc3248) for lock 0x1e260726fbbc0b93 00000080:00010000:2.0:1788572590.336497:0:70355:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b93 from it ffffb68a41c27cd8 00010000:00010000:2.0:1788572590.336501:0:70355:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b93 lrc: 3/1,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4e0f10 expref: -99 pid: 70355 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572590.336509:0:70355:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b93 lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e0f10 expref: -99 pid: 70355 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572590.336559:0:70355:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x9:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788572590.336569:0:70355:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000a811:0x9:0x0] 00000008:00010000:2.0:1788572590.336595:0:70355:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572590.336599:0:70355:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff94d404e6a030, osc lock ffff94d40857b7c0, flags 0x400001002 00000008:00010000:2.0:1788572590.336614:0:70355:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572590.336615:0:70355:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff94d404e6a070, osc lock ffff94d40857a130, flags 0x400001002 00000008:00010000:2.0:1788572590.336632:0:70355:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572590.336634:0:70355:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff94d404e6a0b0, osc lock ffff94d40857b300, flags 0x400001002 00010000:00010000:2.0:1788572590.336651:0:70355:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572590.336663:0:70355:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 1048575) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00000008:00010000:2.0:1788572590.336676:0:70355:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00000020:00200000:2.0:1788572590.336686:0:70355:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788572590.336689:0:70355:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572590.336696:0:70355:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788572590.336712:0:70355:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788572590.336721:0:70355:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (1048576 18446744073709551615) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00000008:00010000:2.0:1788572590.336730:0:70355:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 2097152, setting rss=2097152; kms=2097152, end=18446744073709551615 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00000020:00200000:2.0:1788572590.336743:0:70355:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788572590.336745:0:70355:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 5/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788572590.336753:0:70355:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/0,0 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 7 lru_type: 1 00010000:00010000:2.0:1788572590.336766:0:70355:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x703:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e0a08 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572590.336775:0:70355:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x703:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0xec901abfeb4e0a08 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:2.0:1788572590.336784:0:70355:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x703:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0xec901abfeb4e0a08 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00200000:2.0:1788572590.336792:0:70355:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788572590.336794:0:70355:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x703:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a08 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572590.336802:0:70355:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x703:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a08 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00000020:00200000:2.0:1788572590.336809:0:70355:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788572590.336820:0:70355:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000a811:0x9:0x0] updating i_size 2097152 i_blocks 2048 00000020:00010000:2.0:1788572590.336824:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff94d412026500 (ffff94d405cee488/0) at cl_lock_release():245 00000020:00010000:2.0:1788572590.336827:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d412026500 (ffff94d405cee488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572590.336830:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d404e6a030 (ffff94d405cee1b8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572590.336834:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d404e6a070 (ffff94d405cee098/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572590.336837:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d404e6a0b0 (ffff94d405cee008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572590.336839:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d412026500 (ffff94d405cee488/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572590.336842:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d404e6a030 (ffff94d405cee488/1) at cl_lock_fini():72 00000020:00010000:2.0:1788572590.336845:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d404e6a070 (ffff94d405cee488/1) at cl_lock_fini():72 00000020:00010000:2.0:1788572590.336847:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d404e6a0b0 (ffff94d405cee488/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572590.336885:0:70355:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x9:0x0] ignore/verify layout 0/1, layout version 4 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788572590.336925:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 0, count: 32768 00000080:00200000:2.0:1788572590.336960:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 0 00000080:00200000:2.0:1788572590.336988:0:70355:0:(file.c:2090:ll_file_io_generic()) f132a.replay-single: read ppos: 0, bytes: 32768 00000080:00200000:2.0:1788572590.336993:0:70355:0:(file.c:1848:ll_io_set_mirror()) f132a.replay-single: desiginated mirror: 0 00000020:00200000:2.0:1788572590.336998:0:70355:0:(cl_io.c:214:cl_io_rw_init()) header@0000000099bd4c4d[0x110, 1, [0x20000a811:0x9:0x0] hash] 00000020:00200000:2.0:1788572590.337000:0:70355:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 32768) 0 0 00000080:00200000:2.0:1788572590.337003:0:70355:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572590.337015:0:70355:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 32768) 32768, 32768 00020000:00200000:2.0:1788572590.337019:0:70355:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572590.337039:0:70355:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 7] 00000008:00010000:2.0:1788572590.337047:0:70355:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1788572590.337050:0:70355:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff94d4057bdab0, osc lock ffff94d40857b300, flags 0x0 00010000:00010000:2.0:1788572590.337059:0:70355:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788572590.337068:0:70355:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 32767) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788572590.337077:0:70355:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 6/1,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788572590.337086:0:70355:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 6/1,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788572590.337093:0:70355:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 5/1,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00000080:00200000:2.0:1788572590.337101:0:70355:0:(vvp_io.c:835:vvp_io_read_start()) f132a.replay-single: read [0, 32768) 00000080:00200000:2.0:1788572590.337112:0:70355:0:(vvp_io.c:194:vvp_prep_size()) [0x20000a811:0x9:0x0] updating i_size 2097152 00000080:00200000:2.0:1788572590.337115:0:70355:0:(vvp_io.c:893:vvp_io_read_start()) count:32768 iocb pos:0 00010000:00010000:2.0:1788572590.337148:0:70355:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 4/1,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788572590.337158:0:70355:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (4096 8191) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 4/1,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788572590.337166:0:70355:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 7/2,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788572590.337173:0:70355:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 7/2,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788572590.337180:0:70355:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 6/2,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00000008:00010000:2.0:1788572590.337466:0:70355:0:(osc_cache.c:3010:osc_queue_sync_pages()) ### no dlm lock: extent: ffff94d405c47180 00010000:00010000:2.0:1788572590.337485:0:70355:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 6/2,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788572590.337494:0:70355:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 5/1,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00000008:00010000:2.0:1788572590.337530:0:68734:0:(osc_cache.c:2022:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff94d405c47180 00010000:00010000:2.0:1788572590.337550:0:68734:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 4/1,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00000008:00100000:2.0:1788572590.337593:0:68734:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff94d408c61180 - object 0x280000401:1794 offset 0<>32768 00000100:00100000:3.0:1788572590.338309:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488535680:192.168.202.125@tcp:3:md5sum.0 00000008:00010000:3.0:1788572590.347730:0:2405:0:(osc_cache.c:910:osc_extent_finish()) ### no dlm lock: extent: ffff94d405c47180 00000020:00200000:3.0:1788572590.347743:0:2405:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1788572590.350195:0:2405:0:(osc_cache.c:496:__osc_extent_remove()) ### no dlm lock: extent: ffff94d405c47180 00000100:00100000:3.0:1788572590.350262:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488535680:192.168.202.125@tcp:3:md5sum.0 00000020:00010000:2.0:1788572590.351532:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff94d40392ed58 (ffff94d405cee488/0) at cl_lock_release():245 00000020:00010000:2.0:1788572590.351542:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d40392ed58 (ffff94d405cee488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572590.351546:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d4057bdab0 (ffff94d405cee008/1) at cl_lock_cancel():142 00010000:00010000:2.0:1788572590.351558:0:70355:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 4/1,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788572590.351573:0:70355:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 7 lru_type: 1 00000020:00010000:2.0:1788572590.351588:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d40392ed58 (ffff94d405cee488/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572590.351592:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d4057bdab0 (ffff94d405cee488/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572590.351601:0:70355:0:(vvp_io.c:533:vvp_io_advance()) advancing 32768 bytes 00000080:00200000:2.0:1788572590.351634:0:70355:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x9:0x0] ignore/verify layout 0/0, layout version 4 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572590.351639:0:70355:0:(file.c:2238:ll_file_io_generic()) f132a.replay-single: 1 io complete with rc: 0, result: 32768, restart: 0 00000080:00200000:2.0:1788572590.351644:0:70355:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 32768 00000080:00200008:2.0:1788572590.351739:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 32768, count: 32768 00000080:00200000:2.0:1788572590.352050:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 8 00000080:00200000:2.0:1788572590.352081:0:70355:0:(file.c:2090:ll_file_io_generic()) f132a.replay-single: read ppos: 32768, bytes: 32768 00000080:00200000:2.0:1788572590.352086:0:70355:0:(file.c:1848:ll_io_set_mirror()) f132a.replay-single: desiginated mirror: 0 00000020:00200000:2.0:1788572590.352093:0:70355:0:(cl_io.c:214:cl_io_rw_init()) header@0000000099bd4c4d[0x110, 9, [0x20000a811:0x9:0x0] hash] 00000020:00200000:2.0:1788572590.352094:0:70355:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [32768, 65536) 0 0 00000080:00200000:2.0:1788572590.352099:0:70355:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572590.352114:0:70355:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 32768 chunk: [32768, 65536) 65536, 32768 00020000:00200000:2.0:1788572590.352118:0:70355:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572590.352146:0:70355:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [8, 15] 00000008:00010000:2.0:1788572590.352158:0:70355:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1788572590.352163:0:70355:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff94d4057bdab0, osc lock ffff94d40857b300, flags 0x0 00010000:00010000:2.0:1788572590.352183:0:70355:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 7 lru_type: 1 00010000:00010000:2.0:1788572590.352198:0:70355:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (32768 65535) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 7 lru_type: 1 00010000:00010000:2.0:1788572590.352401:0:70355:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 6/1,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 7 lru_type: 1 00010000:00010000:2.0:1788572590.352410:0:70355:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 6/1,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 7 lru_type: 1 00010000:00010000:2.0:1788572590.352417:0:70355:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 5/1,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 7 lru_type: 1 00000080:00200000:2.0:1788572590.352426:0:70355:0:(vvp_io.c:835:vvp_io_read_start()) f132a.replay-single: read [32768, 65536) 00000080:00200000:2.0:1788572590.352438:0:70355:0:(vvp_io.c:194:vvp_prep_size()) [0x20000a811:0x9:0x0] updating i_size 2097152 00000080:00200000:2.0:1788572590.352442:0:70355:0:(vvp_io.c:893:vvp_io_read_start()) count:32768 iocb pos:32768 00010000:00010000:2.0:1788572590.352470:0:70355:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 4/1,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 7 lru_type: 1 00010000:00010000:2.0:1788572590.352485:0:70355:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 4/1,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 7 lru_type: 1 00010000:00010000:2.0:1788572590.352495:0:70355:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 7/2,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 7 lru_type: 1 00010000:00010000:2.0:1788572590.352503:0:70355:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 7/2,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 7 lru_type: 1 00010000:00010000:2.0:1788572590.352511:0:70355:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 6/2,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 7 lru_type: 1 00010000:00010000:3.0:1788572590.368498:0:70355:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 7 lru_type: 1 00010000:00010000:3.0:1788572590.368524:0:70355:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (1048576 1052671) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 7 lru_type: 1 00010000:00010000:3.0:1788572590.368539:0:70355:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 5/1,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 7 lru_type: 1 00010000:00010000:3.0:1788572590.368547:0:70355:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 5/1,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 7 lru_type: 1 00010000:00010000:3.0:1788572590.368556:0:70355:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/1,0 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 7 lru_type: 1 00000008:00010000:3.0:1788572590.384668:0:70355:0:(osc_cache.c:3010:osc_queue_sync_pages()) ### no dlm lock: extent: ffff94d412408c00 00000008:00010000:3.0:1788572590.384816:0:68887:0:(osc_cache.c:2022:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff94d412408c00 00010000:00010000:3.0:1788572590.384857:0:68887:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (32768 36863) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 6/2,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 7 lru_type: 1 00000008:00100000:3.0:1788572590.384928:0:68887:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff94d408c54a80 - object 0x280000401:1794 offset 32768<>1048576 00000100:00100000:3.0:1788572590.385003:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488535808:192.168.202.125@tcp:3:md5sum.0 00000008:00010000:3.0:1788572590.385257:0:70355:0:(osc_cache.c:3010:osc_queue_sync_pages()) ### no dlm lock: extent: ffff94d4124080c0 00010000:00010000:3.0:1788572590.385272:0:70355:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 6/2,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 7 lru_type: 1 00010000:00010000:3.0:1788572590.385283:0:70355:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 5/1,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 7 lru_type: 1 00010000:00010000:3.0:1788572590.385293:0:70355:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/1,0 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 7 lru_type: 1 00010000:00010000:3.0:1788572590.385303:0:70355:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 8 lru_type: 1 00000008:00010000:3.0:1788572590.385335:0:68887:0:(osc_cache.c:2022:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff94d4124080c0 00010000:00010000:3.0:1788572590.385359:0:68887:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (1048576 1052671) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 9 lru_type: 1 00000008:00100000:3.0:1788572590.385413:0:68887:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff94d408c55500 - object 0x2c0000401:1794 offset 1048576<>2097152 00000100:00100000:3.0:1788572590.385445:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488535936:192.168.202.125@tcp:3:md5sum.0 00000008:00010000:2.0:1788572590.395806:0:2406:0:(osc_cache.c:910:osc_extent_finish()) ### no dlm lock: extent: ffff94d412408c00 00000020:00200000:2.0:1788572590.395834:0:2406:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1788572590.395890:0:2406:0:(osc_cache.c:496:__osc_extent_remove()) ### no dlm lock: extent: ffff94d412408c00 00000100:00100000:2.0:1788572590.395917:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:9bf287c0-f582-4b83-84c1-15e766218620:2406:1875450488535808:192.168.202.125@tcp:3:md5sum.0 00000020:00010000:3.0:1788572590.395954:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff94d40392ed58 (ffff94d405cee488/0) at cl_lock_release():245 00000020:00010000:3.0:1788572590.395959:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d40392ed58 (ffff94d405cee488/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788572590.395966:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d4057bdab0 (ffff94d405cee008/1) at cl_lock_cancel():142 00010000:00010000:3.0:1788572590.395978:0:70355:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 4/1,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 7 lru_type: 1 00010000:00010000:3.0:1788572590.395995:0:70355:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 8 lru_type: 1 00000020:00010000:3.0:1788572590.396016:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d40392ed58 (ffff94d405cee488/0) at cl_lock_fini():72 00000020:00010000:3.0:1788572590.396019:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d4057bdab0 (ffff94d405cee488/1) at cl_lock_fini():72 00000080:00200000:3.0:1788572590.396027:0:70355:0:(vvp_io.c:533:vvp_io_advance()) advancing 32768 bytes 00000080:00200000:3.0:1788572590.396066:0:70355:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x9:0x0] ignore/verify layout 0/0, layout version 4 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572590.396077:0:70355:0:(file.c:2238:ll_file_io_generic()) f132a.replay-single: 1 io complete with rc: 0, result: 32768, restart: 0 00000080:00200000:3.0:1788572590.396083:0:70355:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 32768 00000080:00200008:3.0:1788572590.396180:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 65536, count: 32768 00000080:00200000:3.0:1788572590.396188:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 16 00000080:00200000:3.0:1788572590.396193:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 17 00000080:00200000:3.0:1788572590.396196:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 18 00000080:00200000:3.0:1788572590.396199:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 19 00000080:00200000:3.0:1788572590.396234:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 20 00000080:00200000:3.0:1788572590.396238:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 21 00000080:00200000:3.0:1788572590.396241:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 22 00000080:00200000:3.0:1788572590.396243:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 23 00000080:00200008:2.0:1788572590.397007:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 98304, count: 32768 00000080:00200000:2.0:1788572590.397019:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 24 00000080:00200000:2.0:1788572590.397026:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 25 00000080:00200000:2.0:1788572590.397031:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 26 00000080:00200000:2.0:1788572590.397034:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 27 00000080:00200000:2.0:1788572590.397037:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 28 00000080:00200000:2.0:1788572590.397039:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 29 00000080:00200000:2.0:1788572590.397043:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 30 00000080:00200000:2.0:1788572590.397045:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 31 00000080:00200008:2.0:1788572590.397116:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 131072, count: 32768 00000080:00200000:2.0:1788572590.397121:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 32 00000080:00200000:2.0:1788572590.397125:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 33 00000080:00200000:2.0:1788572590.397128:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 34 00000080:00200000:2.0:1788572590.397131:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 35 00000080:00200000:2.0:1788572590.397134:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 36 00000080:00200000:2.0:1788572590.397137:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 37 00000080:00200000:2.0:1788572590.397140:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 38 00000080:00200000:2.0:1788572590.397143:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 39 00000080:00200008:2.0:1788572590.401090:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 163840, count: 32768 00000080:00200000:2.0:1788572590.401111:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 40 00000080:00200000:2.0:1788572590.401126:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 41 00000080:00200000:2.0:1788572590.401130:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 42 00000080:00200000:2.0:1788572590.401134:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 43 00000080:00200000:2.0:1788572590.401137:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 44 00000080:00200000:2.0:1788572590.401141:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 45 00000080:00200000:2.0:1788572590.401144:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 46 00000080:00200000:2.0:1788572590.401147:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 47 00000080:00200008:2.0:1788572590.401236:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 196608, count: 32768 00000080:00200000:2.0:1788572590.401241:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 48 00000080:00200000:2.0:1788572590.401245:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 49 00000080:00200000:2.0:1788572590.401247:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 50 00000080:00200000:2.0:1788572590.401250:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 51 00000080:00200000:2.0:1788572590.401253:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 52 00000080:00200000:2.0:1788572590.401257:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 53 00000080:00200000:2.0:1788572590.401259:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 54 00000080:00200000:2.0:1788572590.401263:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 55 00000080:00200008:2.0:1788572590.401331:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 229376, count: 32768 00000080:00200000:2.0:1788572590.401336:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 56 00000080:00200000:2.0:1788572590.401340:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 57 00000080:00200000:2.0:1788572590.401343:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 58 00000080:00200000:2.0:1788572590.401346:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 59 00000080:00200000:2.0:1788572590.401348:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 60 00000080:00200000:2.0:1788572590.401351:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 61 00000080:00200000:2.0:1788572590.401354:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 62 00000080:00200000:2.0:1788572590.401357:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 63 00000080:00200008:2.0:1788572590.401426:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 262144, count: 32768 00000080:00200000:2.0:1788572590.401431:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 64 00000080:00200000:2.0:1788572590.401436:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 65 00000080:00200000:2.0:1788572590.401439:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 66 00000080:00200000:2.0:1788572590.401443:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 67 00000080:00200000:2.0:1788572590.401446:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 68 00000080:00200000:2.0:1788572590.401450:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 69 00000080:00200000:2.0:1788572590.401453:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 70 00000080:00200000:2.0:1788572590.401455:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 71 00000080:00200008:2.0:1788572590.401524:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 294912, count: 32768 00000080:00200000:2.0:1788572590.401529:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 72 00000080:00200000:2.0:1788572590.401535:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 73 00000080:00200000:2.0:1788572590.401539:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 74 00000080:00200000:2.0:1788572590.401542:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 75 00000080:00200000:2.0:1788572590.401546:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 76 00000080:00200000:2.0:1788572590.401549:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 77 00000080:00200000:2.0:1788572590.401552:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 78 00000080:00200000:2.0:1788572590.401555:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 79 00000080:00200008:2.0:1788572590.401623:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 327680, count: 32768 00000080:00200000:2.0:1788572590.401628:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 80 00000080:00200000:2.0:1788572590.401632:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 81 00000080:00200000:2.0:1788572590.401636:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 82 00000080:00200000:2.0:1788572590.401638:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 83 00000080:00200000:2.0:1788572590.401641:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 84 00000080:00200000:2.0:1788572590.401645:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 85 00000080:00200000:2.0:1788572590.401648:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 86 00000080:00200000:2.0:1788572590.401651:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 87 00000080:00200008:2.0:1788572590.401786:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 360448, count: 32768 00000080:00200000:2.0:1788572590.401792:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 88 00000080:00200000:2.0:1788572590.401797:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 89 00000080:00200000:2.0:1788572590.401800:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 90 00000080:00200000:2.0:1788572590.401803:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 91 00000080:00200000:2.0:1788572590.401805:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 92 00000080:00200000:2.0:1788572590.401808:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 93 00000080:00200000:2.0:1788572590.401811:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 94 00000080:00200000:2.0:1788572590.401814:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 95 00000080:00200008:2.0:1788572590.401883:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 393216, count: 32768 00000080:00200000:2.0:1788572590.401888:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 96 00000080:00200000:2.0:1788572590.401893:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 97 00000080:00200000:2.0:1788572590.401896:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 98 00000080:00200000:2.0:1788572590.401899:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 99 00000080:00200000:2.0:1788572590.401903:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 100 00000080:00200000:2.0:1788572590.401906:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 101 00000080:00200000:2.0:1788572590.401909:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 102 00000080:00200000:2.0:1788572590.401913:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 103 00000080:00200008:2.0:1788572590.402009:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 425984, count: 32768 00000080:00200000:2.0:1788572590.402014:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 104 00000080:00200000:2.0:1788572590.402019:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 105 00000080:00200000:2.0:1788572590.402022:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 106 00000080:00200000:2.0:1788572590.402026:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 107 00000080:00200000:2.0:1788572590.402030:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 108 00000080:00200000:2.0:1788572590.402033:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 109 00000080:00200000:2.0:1788572590.402037:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 110 00000080:00200000:2.0:1788572590.402040:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 111 00000080:00200008:2.0:1788572590.402109:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 458752, count: 32768 00000080:00200000:2.0:1788572590.402115:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 112 00000080:00200000:2.0:1788572590.402119:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 113 00000080:00200000:2.0:1788572590.402122:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 114 00000080:00200000:2.0:1788572590.402125:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 115 00000080:00200000:2.0:1788572590.402129:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 116 00000080:00200000:2.0:1788572590.402132:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 117 00000080:00200000:2.0:1788572590.402135:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 118 00000080:00200000:2.0:1788572590.402138:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 119 00000080:00200008:2.0:1788572590.405630:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 491520, count: 32768 00000080:00200000:2.0:1788572590.405653:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 120 00000080:00200000:2.0:1788572590.405667:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 121 00000080:00200000:2.0:1788572590.405673:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 122 00000080:00200000:2.0:1788572590.405677:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 123 00000080:00200000:2.0:1788572590.405806:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 124 00000080:00200000:2.0:1788572590.405814:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 125 00000080:00200000:2.0:1788572590.405817:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 126 00000080:00200000:2.0:1788572590.405821:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 127 00000080:00200008:2.0:1788572590.405896:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 524288, count: 32768 00000080:00200000:2.0:1788572590.405902:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 128 00000080:00200000:2.0:1788572590.405907:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 129 00000080:00200000:2.0:1788572590.405910:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 130 00000080:00200000:2.0:1788572590.405913:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 131 00000080:00200000:2.0:1788572590.405916:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 132 00000080:00200000:2.0:1788572590.405919:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 133 00000080:00200000:2.0:1788572590.405923:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 134 00000080:00200000:2.0:1788572590.405926:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 135 00000080:00200008:2.0:1788572590.406006:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 557056, count: 32768 00000080:00200000:2.0:1788572590.406011:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 136 00000080:00200000:2.0:1788572590.406016:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 137 00000080:00200000:2.0:1788572590.406020:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 138 00000080:00200000:2.0:1788572590.406023:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 139 00000080:00200000:2.0:1788572590.406026:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 140 00000080:00200000:2.0:1788572590.406029:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 141 00000080:00200000:2.0:1788572590.406032:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 142 00000080:00200000:2.0:1788572590.406035:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 143 00000080:00200008:2.0:1788572590.406105:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 589824, count: 32768 00000080:00200000:2.0:1788572590.406110:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 144 00000080:00200000:2.0:1788572590.406114:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 145 00000080:00200000:2.0:1788572590.406117:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 146 00000080:00200000:2.0:1788572590.406120:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 147 00000080:00200000:2.0:1788572590.406123:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 148 00000080:00200000:2.0:1788572590.406126:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 149 00000080:00200000:2.0:1788572590.406129:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 150 00000080:00200000:2.0:1788572590.406133:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 151 00000080:00200008:2.0:1788572590.406217:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 622592, count: 32768 00000080:00200000:2.0:1788572590.406223:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 152 00000080:00200000:2.0:1788572590.406227:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 153 00000080:00200000:2.0:1788572590.406229:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 154 00000080:00200000:2.0:1788572590.406231:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 155 00000080:00200000:2.0:1788572590.406234:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 156 00000080:00200000:2.0:1788572590.406237:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 157 00000080:00200000:2.0:1788572590.406240:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 158 00000080:00200000:2.0:1788572590.406252:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 159 00000080:00200008:2.0:1788572590.406322:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 655360, count: 32768 00000080:00200000:2.0:1788572590.406327:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 160 00000080:00200000:2.0:1788572590.406331:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 161 00000080:00200000:2.0:1788572590.406334:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 162 00000080:00200000:2.0:1788572590.406336:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 163 00000080:00200000:2.0:1788572590.406339:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 164 00000080:00200000:2.0:1788572590.406342:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 165 00000080:00200000:2.0:1788572590.406345:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 166 00000080:00200000:2.0:1788572590.406348:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 167 00000080:00200008:2.0:1788572590.406416:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 688128, count: 32768 00000080:00200000:2.0:1788572590.406421:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 168 00000080:00200000:2.0:1788572590.406425:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 169 00000080:00200000:2.0:1788572590.406428:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 170 00000080:00200000:2.0:1788572590.406431:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 171 00000080:00200000:2.0:1788572590.406435:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 172 00000080:00200000:2.0:1788572590.406437:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 173 00000080:00200000:2.0:1788572590.406440:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 174 00000080:00200000:2.0:1788572590.406443:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 175 00000080:00200008:2.0:1788572590.406510:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 720896, count: 32768 00000080:00200000:2.0:1788572590.406515:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 176 00000080:00200000:2.0:1788572590.406519:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 177 00000080:00200000:2.0:1788572590.406522:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 178 00000080:00200000:2.0:1788572590.406526:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 179 00000080:00200000:2.0:1788572590.406529:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 180 00000080:00200000:2.0:1788572590.406532:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 181 00000080:00200000:2.0:1788572590.406536:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 182 00000080:00200000:2.0:1788572590.406539:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 183 00000080:00200008:2.0:1788572590.406607:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 753664, count: 32768 00000080:00200000:2.0:1788572590.406611:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 184 00000080:00200000:2.0:1788572590.406616:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 185 00000080:00200000:2.0:1788572590.406620:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 186 00000080:00200000:2.0:1788572590.406624:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 187 00000080:00200000:2.0:1788572590.406627:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 188 00000080:00200000:2.0:1788572590.406631:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 189 00000080:00200000:2.0:1788572590.406634:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 190 00000080:00200000:2.0:1788572590.406638:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 191 00000080:00200008:2.0:1788572590.407150:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 786432, count: 32768 00000080:00200000:2.0:1788572590.407159:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 192 00000080:00200000:2.0:1788572590.407167:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 193 00000080:00200000:2.0:1788572590.407170:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 194 00000080:00200000:2.0:1788572590.407173:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 195 00000080:00200000:2.0:1788572590.407177:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 196 00000080:00200000:2.0:1788572590.407180:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 197 00000080:00200000:2.0:1788572590.407184:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 198 00000080:00200000:2.0:1788572590.407187:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 199 00000008:00010000:3.0:1788572590.410971:0:2405:0:(osc_cache.c:910:osc_extent_finish()) ### no dlm lock: extent: ffff94d4124080c0 00000008:00010000:3.0:1788572590.411032:0:2405:0:(osc_cache.c:496:__osc_extent_remove()) ### no dlm lock: extent: ffff94d4124080c0 00000100:00100000:3.0:1788572590.411058:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:9bf287c0-f582-4b83-84c1-15e766218620:2405:1875450488535936:192.168.202.125@tcp:3:md5sum.0 00000080:00200008:2.0:1788572590.411577:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 819200, count: 32768 00000080:00200000:2.0:1788572590.411592:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 200 00000080:00200000:2.0:1788572590.411603:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 201 00000080:00200000:2.0:1788572590.411606:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 202 00000080:00200000:2.0:1788572590.411610:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 203 00000080:00200000:2.0:1788572590.411613:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 204 00000080:00200000:2.0:1788572590.411616:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 205 00000080:00200000:2.0:1788572590.411620:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 206 00000080:00200000:2.0:1788572590.411623:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 207 00000080:00200008:2.0:1788572590.411709:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 851968, count: 32768 00000080:00200000:2.0:1788572590.411714:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 208 00000080:00200000:2.0:1788572590.411719:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 209 00000080:00200000:2.0:1788572590.411722:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 210 00000080:00200000:2.0:1788572590.411725:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 211 00000080:00200000:2.0:1788572590.411728:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 212 00000080:00200000:2.0:1788572590.411731:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 213 00000080:00200000:2.0:1788572590.411734:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 214 00000080:00200000:2.0:1788572590.411737:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 215 00000080:00200008:2.0:1788572590.411805:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 884736, count: 32768 00000080:00200000:2.0:1788572590.411811:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 216 00000080:00200000:2.0:1788572590.411815:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 217 00000080:00200000:2.0:1788572590.411819:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 218 00000080:00200000:2.0:1788572590.411822:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 219 00000080:00200000:2.0:1788572590.411825:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 220 00000080:00200000:2.0:1788572590.411828:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 221 00000080:00200000:2.0:1788572590.411830:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 222 00000080:00200000:2.0:1788572590.411834:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 223 00000080:00200008:2.0:1788572590.411901:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 917504, count: 32768 00000080:00200000:2.0:1788572590.411906:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 224 00000080:00200000:2.0:1788572590.411911:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 225 00000080:00200000:2.0:1788572590.411914:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 226 00000080:00200000:2.0:1788572590.411918:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 227 00000080:00200000:2.0:1788572590.411922:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 228 00000080:00200000:2.0:1788572590.411926:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 229 00000080:00200000:2.0:1788572590.411940:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 230 00000080:00200000:2.0:1788572590.411944:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 231 00000080:00200008:2.0:1788572590.412014:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 950272, count: 32768 00000080:00200000:2.0:1788572590.412019:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 232 00000080:00200000:2.0:1788572590.412023:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 233 00000080:00200000:2.0:1788572590.412027:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 234 00000080:00200000:2.0:1788572590.412030:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 235 00000080:00200000:2.0:1788572590.412033:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 236 00000080:00200000:2.0:1788572590.412036:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 237 00000080:00200000:2.0:1788572590.412039:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 238 00000080:00200000:2.0:1788572590.412042:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 239 00000080:00200008:2.0:1788572590.412111:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 983040, count: 32768 00000080:00200000:2.0:1788572590.412116:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 240 00000080:00200000:2.0:1788572590.412120:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 241 00000080:00200000:2.0:1788572590.412123:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 242 00000080:00200000:2.0:1788572590.412126:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 243 00000080:00200000:2.0:1788572590.412130:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 244 00000080:00200000:2.0:1788572590.412133:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 245 00000080:00200000:2.0:1788572590.412136:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 246 00000080:00200000:2.0:1788572590.412139:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 247 00000080:00200008:2.0:1788572590.412227:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1015808, count: 32768 00000080:00200000:2.0:1788572590.412232:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 248 00000080:00200000:2.0:1788572590.412236:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 249 00000080:00200000:2.0:1788572590.412239:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 250 00000080:00200000:2.0:1788572590.412242:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 251 00000080:00200000:2.0:1788572590.412245:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 252 00000080:00200000:2.0:1788572590.412248:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 253 00000080:00200000:2.0:1788572590.412250:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 254 00000080:00200000:2.0:1788572590.412253:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 255 00000080:00200008:2.0:1788572590.412321:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1048576, count: 32768 00000080:00200000:2.0:1788572590.412326:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 256 00000080:00200000:2.0:1788572590.412330:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 257 00000080:00200000:2.0:1788572590.412333:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 258 00000080:00200000:2.0:1788572590.412336:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 259 00000080:00200000:2.0:1788572590.412339:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 260 00000080:00200000:2.0:1788572590.412342:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 261 00000080:00200000:2.0:1788572590.412345:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 262 00000080:00200000:2.0:1788572590.412348:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 263 00000080:00200008:2.0:1788572590.412416:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1081344, count: 32768 00000080:00200000:2.0:1788572590.412422:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 264 00000080:00200000:2.0:1788572590.412426:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 265 00000080:00200000:2.0:1788572590.412429:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 266 00000080:00200000:2.0:1788572590.412433:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 267 00000080:00200000:2.0:1788572590.412436:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 268 00000080:00200000:2.0:1788572590.412439:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 269 00000080:00200000:2.0:1788572590.412442:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 270 00000080:00200000:2.0:1788572590.412445:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 271 00000080:00200008:2.0:1788572590.412513:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1114112, count: 32768 00000080:00200000:2.0:1788572590.412517:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 272 00000080:00200000:2.0:1788572590.412521:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 273 00000080:00200000:2.0:1788572590.412524:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 274 00000080:00200000:2.0:1788572590.412527:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 275 00000080:00200000:2.0:1788572590.412530:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 276 00000080:00200000:2.0:1788572590.412532:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 277 00000080:00200000:2.0:1788572590.412535:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 278 00000080:00200000:2.0:1788572590.412538:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 279 00000080:00200008:2.0:1788572590.412609:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1146880, count: 32768 00000080:00200000:2.0:1788572590.412612:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 280 00000080:00200000:2.0:1788572590.412615:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 281 00000080:00200000:2.0:1788572590.412618:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 282 00000080:00200000:2.0:1788572590.412621:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 283 00000080:00200000:2.0:1788572590.412624:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 284 00000080:00200000:2.0:1788572590.412626:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 285 00000080:00200000:2.0:1788572590.412629:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 286 00000080:00200000:2.0:1788572590.412632:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 287 00000080:00200008:2.0:1788572590.412698:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1179648, count: 32768 00000080:00200000:2.0:1788572590.412702:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 288 00000080:00200000:2.0:1788572590.412705:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 289 00000080:00200000:2.0:1788572590.412708:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 290 00000080:00200000:2.0:1788572590.412711:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 291 00000080:00200000:2.0:1788572590.412713:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 292 00000080:00200000:2.0:1788572590.412716:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 293 00000080:00200000:2.0:1788572590.412720:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 294 00000080:00200000:2.0:1788572590.412723:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 295 00000080:00200008:2.0:1788572590.412791:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1212416, count: 32768 00000080:00200000:2.0:1788572590.412796:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 296 00000080:00200000:2.0:1788572590.412800:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 297 00000080:00200000:2.0:1788572590.412803:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 298 00000080:00200000:2.0:1788572590.412805:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 299 00000080:00200000:2.0:1788572590.412809:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 300 00000080:00200000:2.0:1788572590.412812:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 301 00000080:00200000:2.0:1788572590.412814:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 302 00000080:00200000:2.0:1788572590.412817:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 303 00000080:00200008:2.0:1788572590.412885:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1245184, count: 32768 00000080:00200000:2.0:1788572590.412889:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 304 00000080:00200000:2.0:1788572590.412893:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 305 00000080:00200000:2.0:1788572590.412896:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 306 00000080:00200000:2.0:1788572590.412898:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 307 00000080:00200000:2.0:1788572590.412901:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 308 00000080:00200000:2.0:1788572590.412903:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 309 00000080:00200000:2.0:1788572590.412907:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 310 00000080:00200000:2.0:1788572590.412909:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 311 00000080:00200008:2.0:1788572590.412989:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1277952, count: 32768 00000080:00200000:2.0:1788572590.412994:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 312 00000080:00200000:2.0:1788572590.412998:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 313 00000080:00200000:2.0:1788572590.413001:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 314 00000080:00200000:2.0:1788572590.413004:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 315 00000080:00200000:2.0:1788572590.413007:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 316 00000080:00200000:2.0:1788572590.413010:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 317 00000080:00200000:2.0:1788572590.413013:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 318 00000080:00200000:2.0:1788572590.413015:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 319 00000080:00200008:2.0:1788572590.413084:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1310720, count: 32768 00000080:00200000:2.0:1788572590.413088:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 320 00000080:00200000:2.0:1788572590.413092:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 321 00000080:00200000:2.0:1788572590.413095:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 322 00000080:00200000:2.0:1788572590.413098:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 323 00000080:00200000:2.0:1788572590.413101:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 324 00000080:00200000:2.0:1788572590.413104:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 325 00000080:00200000:2.0:1788572590.413107:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 326 00000080:00200000:2.0:1788572590.413110:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 327 00000080:00200008:2.0:1788572590.413178:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1343488, count: 32768 00000080:00200000:2.0:1788572590.413183:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 328 00000080:00200000:2.0:1788572590.413188:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 329 00000080:00200000:2.0:1788572590.413191:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 330 00000080:00200000:2.0:1788572590.413194:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 331 00000080:00200000:2.0:1788572590.413197:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 332 00000080:00200000:2.0:1788572590.413199:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 333 00000080:00200000:2.0:1788572590.413223:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 334 00000080:00200000:2.0:1788572590.413227:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 335 00000080:00200008:2.0:1788572590.413298:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1376256, count: 32768 00000080:00200000:2.0:1788572590.413303:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 336 00000080:00200000:2.0:1788572590.413307:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 337 00000080:00200000:2.0:1788572590.413310:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 338 00000080:00200000:2.0:1788572590.413313:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 339 00000080:00200000:2.0:1788572590.413316:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 340 00000080:00200000:2.0:1788572590.413319:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 341 00000080:00200000:2.0:1788572590.413322:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 342 00000080:00200000:2.0:1788572590.413325:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 343 00000080:00200008:2.0:1788572590.413394:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1409024, count: 32768 00000080:00200000:2.0:1788572590.413398:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 344 00000080:00200000:2.0:1788572590.413402:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 345 00000080:00200000:2.0:1788572590.413404:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 346 00000080:00200000:2.0:1788572590.413408:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 347 00000080:00200000:2.0:1788572590.413410:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 348 00000080:00200000:2.0:1788572590.413413:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 349 00000080:00200000:2.0:1788572590.413416:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 350 00000080:00200000:2.0:1788572590.413419:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 351 00000080:00200008:2.0:1788572590.413485:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1441792, count: 32768 00000080:00200000:2.0:1788572590.413488:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 352 00000080:00200000:2.0:1788572590.413492:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 353 00000080:00200000:2.0:1788572590.413495:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 354 00000080:00200000:2.0:1788572590.413498:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 355 00000080:00200000:2.0:1788572590.413501:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 356 00000080:00200000:2.0:1788572590.413504:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 357 00000080:00200000:2.0:1788572590.413506:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 358 00000080:00200000:2.0:1788572590.413509:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 359 00000080:00200008:2.0:1788572590.413576:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1474560, count: 32768 00000080:00200000:2.0:1788572590.413581:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 360 00000080:00200000:2.0:1788572590.413585:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 361 00000080:00200000:2.0:1788572590.413588:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 362 00000080:00200000:2.0:1788572590.413591:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 363 00000080:00200000:2.0:1788572590.413594:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 364 00000080:00200000:2.0:1788572590.413597:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 365 00000080:00200000:2.0:1788572590.413599:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 366 00000080:00200000:2.0:1788572590.413602:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 367 00000080:00200008:2.0:1788572590.413669:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1507328, count: 32768 00000080:00200000:2.0:1788572590.413674:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 368 00000080:00200000:2.0:1788572590.413677:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 369 00000080:00200000:2.0:1788572590.413680:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 370 00000080:00200000:2.0:1788572590.413683:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 371 00000080:00200000:2.0:1788572590.413686:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 372 00000080:00200000:2.0:1788572590.413688:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 373 00000080:00200000:2.0:1788572590.413691:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 374 00000080:00200000:2.0:1788572590.413694:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 375 00000080:00200008:2.0:1788572590.413762:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1540096, count: 32768 00000080:00200000:2.0:1788572590.413768:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 376 00000080:00200000:2.0:1788572590.413772:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 377 00000080:00200000:2.0:1788572590.413775:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 378 00000080:00200000:2.0:1788572590.413778:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 379 00000080:00200000:2.0:1788572590.413781:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 380 00000080:00200000:2.0:1788572590.413785:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 381 00000080:00200000:2.0:1788572590.413788:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 382 00000080:00200000:2.0:1788572590.413791:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 383 00000080:00200008:2.0:1788572590.413860:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1572864, count: 32768 00000080:00200000:2.0:1788572590.413865:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 384 00000080:00200000:2.0:1788572590.413869:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 385 00000080:00200000:2.0:1788572590.413872:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 386 00000080:00200000:2.0:1788572590.413875:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 387 00000080:00200000:2.0:1788572590.413877:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 388 00000080:00200000:2.0:1788572590.413880:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 389 00000080:00200000:2.0:1788572590.413883:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 390 00000080:00200000:2.0:1788572590.413886:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 391 00000080:00200008:2.0:1788572590.413966:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1605632, count: 32768 00000080:00200000:2.0:1788572590.413970:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 392 00000080:00200000:2.0:1788572590.413975:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 393 00000080:00200000:2.0:1788572590.413979:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 394 00000080:00200000:2.0:1788572590.413982:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 395 00000080:00200000:2.0:1788572590.413985:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 396 00000080:00200000:2.0:1788572590.413988:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 397 00000080:00200000:2.0:1788572590.413991:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 398 00000080:00200000:2.0:1788572590.413994:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 399 00000080:00200008:2.0:1788572590.414063:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1638400, count: 32768 00000080:00200000:2.0:1788572590.414068:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 400 00000080:00200000:2.0:1788572590.414072:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 401 00000080:00200000:2.0:1788572590.414075:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 402 00000080:00200000:2.0:1788572590.414079:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 403 00000080:00200000:2.0:1788572590.414081:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 404 00000080:00200000:2.0:1788572590.414084:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 405 00000080:00200000:2.0:1788572590.414087:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 406 00000080:00200000:2.0:1788572590.414091:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 407 00000080:00200008:2.0:1788572590.414160:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1671168, count: 32768 00000080:00200000:2.0:1788572590.414164:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 408 00000080:00200000:2.0:1788572590.414169:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 409 00000080:00200000:2.0:1788572590.414172:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 410 00000080:00200000:2.0:1788572590.414175:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 411 00000080:00200000:2.0:1788572590.414178:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 412 00000080:00200000:2.0:1788572590.414181:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 413 00000080:00200000:2.0:1788572590.414184:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 414 00000080:00200000:2.0:1788572590.414187:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 415 00000080:00200008:2.0:1788572590.414276:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1703936, count: 32768 00000080:00200000:2.0:1788572590.414281:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 416 00000080:00200000:2.0:1788572590.414285:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 417 00000080:00200000:2.0:1788572590.414288:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 418 00000080:00200000:2.0:1788572590.414292:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 419 00000080:00200000:2.0:1788572590.414295:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 420 00000080:00200000:2.0:1788572590.414297:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 421 00000080:00200000:2.0:1788572590.414300:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 422 00000080:00200000:2.0:1788572590.414303:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 423 00000080:00200008:2.0:1788572590.414372:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1736704, count: 32768 00000080:00200000:2.0:1788572590.414376:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 424 00000080:00200000:2.0:1788572590.414380:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 425 00000080:00200000:2.0:1788572590.414383:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 426 00000080:00200000:2.0:1788572590.414386:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 427 00000080:00200000:2.0:1788572590.414389:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 428 00000080:00200000:2.0:1788572590.414392:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 429 00000080:00200000:2.0:1788572590.414395:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 430 00000080:00200000:2.0:1788572590.414398:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 431 00000080:00200008:2.0:1788572590.414465:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1769472, count: 32768 00000080:00200000:2.0:1788572590.414469:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 432 00000080:00200000:2.0:1788572590.414473:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 433 00000080:00200000:2.0:1788572590.414476:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 434 00000080:00200000:2.0:1788572590.414479:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 435 00000080:00200000:2.0:1788572590.414481:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 436 00000080:00200000:2.0:1788572590.414484:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 437 00000080:00200000:2.0:1788572590.414487:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 438 00000080:00200000:2.0:1788572590.414489:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 439 00000080:00200008:2.0:1788572590.414559:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1802240, count: 32768 00000080:00200000:2.0:1788572590.414563:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 440 00000080:00200000:2.0:1788572590.414567:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 441 00000080:00200000:2.0:1788572590.414570:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 442 00000080:00200000:2.0:1788572590.414573:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 443 00000080:00200000:2.0:1788572590.414576:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 444 00000080:00200000:2.0:1788572590.414579:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 445 00000080:00200000:2.0:1788572590.414581:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 446 00000080:00200000:2.0:1788572590.414585:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 447 00000080:00200008:2.0:1788572590.414651:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1835008, count: 32768 00000080:00200000:2.0:1788572590.414656:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 448 00000080:00200000:2.0:1788572590.414660:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 449 00000080:00200000:2.0:1788572590.414663:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 450 00000080:00200000:2.0:1788572590.414666:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 451 00000080:00200000:2.0:1788572590.414669:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 452 00000080:00200000:2.0:1788572590.414672:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 453 00000080:00200000:2.0:1788572590.414674:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 454 00000080:00200000:2.0:1788572590.414677:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 455 00000080:00200008:2.0:1788572590.414745:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1867776, count: 32768 00000080:00200000:2.0:1788572590.414749:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 456 00000080:00200000:2.0:1788572590.414754:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 457 00000080:00200000:2.0:1788572590.414757:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 458 00000080:00200000:2.0:1788572590.414760:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 459 00000080:00200000:2.0:1788572590.414763:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 460 00000080:00200000:2.0:1788572590.414766:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 461 00000080:00200000:2.0:1788572590.414769:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 462 00000080:00200000:2.0:1788572590.414771:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 463 00000080:00200008:2.0:1788572590.414872:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1900544, count: 32768 00000080:00200000:2.0:1788572590.414877:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 464 00000080:00200000:2.0:1788572590.414881:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 465 00000080:00200000:2.0:1788572590.414884:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 466 00000080:00200000:2.0:1788572590.414888:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 467 00000080:00200000:2.0:1788572590.414891:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 468 00000080:00200000:2.0:1788572590.414894:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 469 00000080:00200000:2.0:1788572590.414898:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 470 00000080:00200000:2.0:1788572590.414901:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 471 00000080:00200008:2.0:1788572590.420434:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1933312, count: 32768 00000080:00200000:2.0:1788572590.422015:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 472 00000080:00200000:2.0:1788572590.422035:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 473 00000080:00200000:2.0:1788572590.422040:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 474 00000080:00200000:2.0:1788572590.422044:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 475 00000080:00200000:2.0:1788572590.422049:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 476 00000080:00200000:2.0:1788572590.422053:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 477 00000080:00200000:2.0:1788572590.422057:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 478 00000080:00200000:2.0:1788572590.422061:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 479 00000080:00200008:2.0:1788572590.422147:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1966080, count: 32768 00000080:00200000:2.0:1788572590.422155:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 480 00000080:00200000:2.0:1788572590.422159:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 481 00000080:00200000:2.0:1788572590.422163:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 482 00000080:00200000:2.0:1788572590.422166:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 483 00000080:00200000:2.0:1788572590.422169:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 484 00000080:00200000:2.0:1788572590.422172:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 485 00000080:00200000:2.0:1788572590.422175:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 486 00000080:00200000:2.0:1788572590.422178:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 487 00000080:00200008:2.0:1788572590.422262:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 1998848, count: 32768 00000080:00200000:2.0:1788572590.422268:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 488 00000080:00200000:2.0:1788572590.422272:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 489 00000080:00200000:2.0:1788572590.422275:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 490 00000080:00200000:2.0:1788572590.422278:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 491 00000080:00200000:2.0:1788572590.422281:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 492 00000080:00200000:2.0:1788572590.422284:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 493 00000080:00200000:2.0:1788572590.422286:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 494 00000080:00200000:2.0:1788572590.422289:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 495 00000080:00200008:2.0:1788572590.422358:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 2031616, count: 32768 00000080:00200000:2.0:1788572590.422362:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 496 00000080:00200000:2.0:1788572590.422366:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 497 00000080:00200000:2.0:1788572590.422368:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 498 00000080:00200000:2.0:1788572590.422369:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 499 00000080:00200000:2.0:1788572590.422371:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 500 00000080:00200000:2.0:1788572590.422373:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 501 00000080:00200000:2.0:1788572590.422375:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 502 00000080:00200000:2.0:1788572590.422377:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 503 00000080:00200008:2.0:1788572590.422439:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 2064384, count: 32768 00000080:00200000:2.0:1788572590.422441:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 504 00000080:00200000:2.0:1788572590.422444:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 505 00000080:00200000:2.0:1788572590.422446:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 506 00000080:00200000:2.0:1788572590.422448:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 507 00000080:00200000:2.0:1788572590.422450:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 508 00000080:00200000:2.0:1788572590.422452:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 509 00000080:00200000:2.0:1788572590.422455:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 510 00000080:00200000:2.0:1788572590.422457:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 511 00000080:00200008:2.0:1788572590.422519:0:70355:0:(file.c:2483:do_file_read_iter()) START file f132a.replay-single:[0x20000a811:0x9:0x0], ppos: 2097152, count: 32768 00000080:00200000:2.0:1788572590.422554:0:70355:0:(rw.c:1925:ll_readpage()) fast read pgno: 512 00000080:00200000:2.0:1788572590.422584:0:70355:0:(file.c:2090:ll_file_io_generic()) f132a.replay-single: read ppos: 2097152, bytes: 32768 00000080:00200000:2.0:1788572590.422588:0:70355:0:(file.c:1848:ll_io_set_mirror()) f132a.replay-single: desiginated mirror: 0 00000020:00200000:2.0:1788572590.422594:0:70355:0:(cl_io.c:214:cl_io_rw_init()) header@0000000099bd4c4d[0x110, 513, [0x20000a811:0x9:0x0] hash] 00000020:00200000:2.0:1788572590.422596:0:70355:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [2097152, 2129920) 0 0 00000080:00200000:2.0:1788572590.422600:0:70355:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572590.422616:0:70355:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [2097152, 2129920) 2129920, 32768 00020000:00200000:2.0:1788572590.422619:0:70355:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:2.0:1788572590.422640:0:70355:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [512, 519] 00000008:00010000:2.0:1788572590.422650:0:70355:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1788572590.422654:0:70355:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff94d4057bdf30, osc lock ffff94d40857a130, flags 0x0 00010000:00010000:2.0:1788572590.422672:0:70355:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788572590.422755:0:70355:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (2097152 2129919) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788572590.422769:0:70355:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 6/1,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788572590.422776:0:70355:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 6/1,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788572590.422782:0:70355:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 5/1,0 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 9 lru_type: 1 00000080:00200000:2.0:1788572590.422788:0:70355:0:(vvp_io.c:835:vvp_io_read_start()) f132a.replay-single: read [2097152, 2129920) 00000001:00010000:2.0:1788572590.422801:0:70355:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000a811:0x9:0x0] 00000008:00010000:2.0:1788572590.422816:0:70355:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572590.422817:0:70355:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff94d404e6a030, osc lock ffff94d40857b7c0, flags 0x400001002 00000008:00010000:2.0:1788572590.422820:0:70355:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572590.422821:0:70355:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff94d404e6a070, osc lock ffff94d40857b300, flags 0x400001002 00000008:00010000:2.0:1788572590.422831:0:70355:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572590.422832:0:70355:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff94d404e6a0b0, osc lock ffff94d40857b430, flags 0x400001002 00010000:00010000:2.0:1788572590.422840:0:70355:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 8 lru_type: 1 00010000:00010000:2.0:1788572590.422847:0:70355:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 1048575) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 8 lru_type: 1 00000020:00200000:2.0:1788572590.422852:0:70355:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788572590.422854:0:70355:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 8 lru_type: 1 00010000:00010000:2.0:1788572590.422859:0:70355:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788572590.422873:0:70355:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/1,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788572590.422878:0:70355:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (1048576 18446744073709551615) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/1,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 9 lru_type: 1 00000020:00200000:2.0:1788572590.422882:0:70355:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788572590.422884:0:70355:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 7/1,1 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788572590.422888:0:70355:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 6/1,0 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788572590.422893:0:70355:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x703:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a08 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572590.422897:0:70355:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x703:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a08 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00000020:00200000:2.0:1788572590.422901:0:70355:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788572590.422902:0:70355:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x703:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a08 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572590.422906:0:70355:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x703:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a08 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00000020:00200000:2.0:1788572590.422911:0:70355:0:(cl_io.c:1862:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788572590.422916:0:70355:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000a811:0x9:0x0] updating i_size 2097152 i_blocks 2048 00000020:00010000:2.0:1788572590.422919:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff94d412024de0 (ffff94d405cee098/0) at cl_lock_release():245 00000020:00010000:2.0:1788572590.422921:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d412024de0 (ffff94d405cee098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572590.422922:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d404e6a030 (ffff94d405cee5a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572590.422925:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d404e6a070 (ffff94d405cee1b8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572590.422927:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d404e6a0b0 (ffff94d405cee998/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572590.422940:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d412024de0 (ffff94d405cee098/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572590.422942:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d404e6a030 (ffff94d405cee098/1) at cl_lock_fini():72 00000020:00010000:2.0:1788572590.422948:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d404e6a070 (ffff94d405cee098/1) at cl_lock_fini():72 00000020:00010000:2.0:1788572590.422950:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d404e6a0b0 (ffff94d405cee098/1) at cl_lock_fini():72 00000020:00010000:2.0:1788572590.422962:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff94d40392ef38 (ffff94d405cee098/0) at cl_lock_release():245 00000020:00010000:2.0:1788572590.422964:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d40392ef38 (ffff94d405cee098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572590.422967:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff94d4057bdf30 (ffff94d405cee1b8/1) at cl_lock_cancel():142 00010000:00010000:2.0:1788572590.422971:0:70355:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 4/1,0 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788572590.422979:0:70355:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 10 lru_type: 1 00000020:00010000:2.0:1788572590.422987:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d40392ef38 (ffff94d405cee098/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572590.422990:0:70355:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff94d4057bdf30 (ffff94d405cee098/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572590.422995:0:70355:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000080:00200000:2.0:1788572590.423040:0:70355:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x9:0x0] ignore/verify layout 0/0, layout version 4 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572590.423044:0:70355:0:(file.c:2238:ll_file_io_generic()) f132a.replay-single: 1 io complete with rc: 0, result: 0, restart: 0 00000080:00200000:2.0:1788572590.423048:0:70355:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 0 00000080:00200008:2.0:1788572590.423063:0:70355:0:(file.c:5203:ll_file_seek()) START file f132a.replay-single:[0x20000a811:0x9:0x0], offset: 0, type: SEEK_CUR 00000080:00200008:2.0:1788572590.423069:0:70355:0:(file.c:5234:ll_file_seek()) COMPLETED file f132a.replay-single:[0x20000a811:0x9:0x0], offset: 0, type: SEEK_CUR, rc = 2097152 00000080:00200008:2.0:1788572590.423085:0:70355:0:(file.c:479:ll_file_release()) START file f132a.replay-single:[0x20000a811:0x9:0x0](ffff94d410bc3248), flags 100000 00000002:00100000:2.0:1788572590.423377:0:70355:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff94d408d3e300 x1875450488535552/t111669149697(111669149697) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 800/872 e 0 to 0 dl 1788572606 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'md5sum.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572590.423389:0:70355:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572590.423394:0:70355:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408c62680 x1875450488536192/t0(0) o35->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/760 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572590.423415:0:70355:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job md5sum:9bf287c0-f582-4b83-84c1-15e766218620:70355:1875450488536192:192.168.202.125@tcp:35:md5sum.0 00000100:00100000:2.0:1788572590.423491:0:70355:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405388f00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572590.426020:0:70355:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job md5sum:9bf287c0-f582-4b83-84c1-15e766218620:70355:1875450488536192:192.168.202.125@tcp:35:md5sum.0 00000002:00100000:2.0:1788572590.426057:0:70355:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff94d408d3e300 x1875450488535552/t111669149697(111669149697) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 800/872 e 0 to 0 dl 1788572606 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'md5sum.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1788572590.725294:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572591.749362:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572592.774643:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572593.800809:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572594.828456:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572595.401257:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488536448:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572595.401344:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488536704:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572595.401376:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:1.0:1788572595.401382:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488536320:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572595.401409:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488536576:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572595.401426:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488536832:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572595.412450:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488536320:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572595.415400:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488536576:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572595.415418:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488536832:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572595.415429:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488536448:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572595.415439:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9bf287c0-f582-4b83-84c1-15e766218620:2404:1875450488536704:192.168.202.125@tcp:400:kworker.0 00010000:00010000:2.0:1788572595.851388:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1788572596.478121:0:70924:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 133: check resend of ongoing requests for lwp during failover ========================================================== 21:43:15 (1788572595) 00000080:00200000:0.0:1788572596.672117:0:70937:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:0.0:1788572596.672148:0:70937:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572596.672166:0:70937:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:0.0:1788572596.672183:0:70937:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:0.0:1788572596.672192:0:70937:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572596.672199:0:70937:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572596.672477:0:70937:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0b7e 00000080:00010000:0.0:1788572596.672486:0:70937:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b7e from it ffffb68a41c47e18 00010000:00010000:0.0:1788572596.672491:0:70937:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:0.0:1788572596.672501:0:70937:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:0.0:1788572596.672519:0:70937:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 101 00000080:00200000:0.0:1788572596.672530:0:70937:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:0.0:1788572596.672538:0:70937:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572596.672616:0:70937:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:0.0:1788572596.672627:0:70937:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:0.0:1788572596.672634:0:70937:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572596.672638:0:70937:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572596.672640:0:70937:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0b7e 00000080:00010000:0.0:1788572596.672644:0:70937:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b7e from it ffffb68a41c47be0 00010000:00010000:0.0:1788572596.672647:0:70937:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:0.0:1788572596.672654:0:70937:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:0.0:1788572596.672662:0:70937:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572596.672673:0:70937:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d133.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), flags=1536 00000080:00200000:0.0:1788572596.672677:0:70937:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:0.0:1788572596.672905:0:70937:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572596.672913:0:70937:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:0.0:1788572596.672927:0:70937:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:0.0:1788572596.672934:0:70937:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572596.672939:0:70937:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572596.672941:0:70937:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0b7e 00000080:00010000:0.0:1788572596.672945:0:70937:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b7e from it ffffb68a41c47ce0 00010000:00010000:0.0:1788572596.672949:0:70937:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:0.0:1788572596.672956:0:70937:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:0.0:1788572596.672965:0:70937:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 3 00000080:00200000:0.0:1788572596.672971:0:70937:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:0.0:1788572596.672977:0:70937:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572596.672983:0:70937:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:0.0:1788572596.672991:0:70937:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:0.0:1788572596.672997:0:70937:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572596.673001:0:70937:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572596.673003:0:70937:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0b7e 00000080:00010000:0.0:1788572596.673007:0:70937:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b7e from it ffffb68a41c47dc8 00010000:00010000:0.0:1788572596.673011:0:70937:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:0.0:1788572596.673018:0:70937:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:0.0:1788572596.673025:0:70937:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 3 00000080:00200000:0.0:1788572596.673031:0:70937:0:(namei.c:2239:do_mkdir()) VFS Op:name=d133.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988) 00000020:00100000:0.0:1788572596.673097:0:70937:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788572596.673110:0:70937:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d408d3f480 x1875450488537088/t0(0) o36->lustre-MDT0001-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 568/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572596.673135:0:70937:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job mkdir:9bf287c0-f582-4b83-84c1-15e766218620:70937:1875450488537088:192.168.202.125@tcp:36:mkdir.0 00000100:00100000:0.0:1788572596.673344:0:70937:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405411600 going to sleep for 11 seconds 00000100:00100000:3.0:1788572596.678884:0:2397:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.125@tcp (source: 12345-192.168.202.125@tcp) 00000100:00100000:2.0:1788572596.678950:0:42440:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875450511553408 00000100:00100000:2.0:1788572596.678959:0:42440:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875450511553408 00000100:00100000:2.0:1788572596.679091:0:42440:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.125@tcp, seq: 96 00000100:00100000:2.0:1788572596.679102:0:42440:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff94d408c55500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:9bf287c0-f582-4b83-84c1-15e766218620+5:8443:x1875450511553408:12345-192.168.202.125@tcp:104: 00010000:00010000:2.0:1788572596.679119:0:42440:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:2.0:1788572596.679221:0:42440:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788572596.679228:0:42440:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff94d408c55500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:9bf287c0-f582-4b83-84c1-15e766218620+5:8443:x1875450511553408:12345-192.168.202.125@tcp:104: Request processed in 128us (353us total) trans 0 rc 0/0 00000100:00100000:2.0:1788572596.679238:0:42440:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.125@tcp, seq: 96 00010000:00010000:0.0:1788572596.679295:0:42448:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788572596.679302:0:42448:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:0.0:1788572596.679311:0:42448:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412537000 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:0.0:1788572596.679316:0:42448:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:0.0:1788572596.679325:0:42448:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:0.0:1788572596.679334:0:42448:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:0.0:1788572596.679385:0:42448:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:0.0:1788572596.679393:0:42448:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572596.679420:0:42448:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:0.0:1788572596.679429:0:42448:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412537000/0x1e260726fbbc0b7e lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xec901abfeb4e0ea0 expref: -99 pid: 69862 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:0.0:1788572596.679445:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572596.679447:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572596.679448:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572596.679455:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1788572596.679479:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488537216:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572596.681303:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:9bf287c0-f582-4b83-84c1-15e766218620:2403:1875450488537216:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572596.692396:0:70937:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job mkdir:9bf287c0-f582-4b83-84c1-15e766218620:70937:1875450488537088:192.168.202.125@tcp:36:mkdir.0 00000080:00200000:2.0:1788572596.692489:0:70937:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x240006991:0xc:0x0](ffff94d403436c08) 00000080:00200000:2.0:1788572596.692495:0:70937:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x240006991:0xc:0x0], updating i_size 4096 00000080:00200000:2.0:1788572596.698427:0:70938:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572596.698452:0:70938:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572596.698499:0:70938:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572596.698507:0:70938:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572596.698514:0:70938:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572596.698534:0:70938:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70938:1875450488537344:192.168.202.125@tcp:101:lfs.0 00000100:00100000:2.0:1788572596.698670:0:70938:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d407e13480 going to sleep for 11 seconds 00000100:00100000:0.0:1788572596.700872:0:70938:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 111669149699 gen 4 00000100:00100000:0.0:1788572596.700889:0:70938:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70938:1875450488537344:192.168.202.125@tcp:101:lfs.0 00010000:00010000:0.0:1788572596.700909:0:70938:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572596.700921:0:70938:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572596.700934:0:70938:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572596.700948:0:70938:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572596.700960:0:70938:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572596.700967:0:70938:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572596.700974:0:70938:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572596.700984:0:70938:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d408c62680 x1875450488537344/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572612 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572596.700996:0:70938:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572596.701008:0:70938:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788572596.701014:0:70938:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788572596.701040:0:70938:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1788572596.701045:0:70938:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0b9a 00000080:00010000:0.0:1788572596.701050:0:70938:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:0.0:1788572596.701053:0:70938:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b9a from it ffffb68a41ff7b58 00010000:00010000:0.0:1788572596.701056:0:70938:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572596.701068:0:70938:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572596.701090:0:70938:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572596.701108:0:70938:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d133.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), flags=29 00000080:00200000:0.0:1788572596.701113:0:70938:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d133.replay-single, dir=[0x200000007:0x1:0x0](ffff94d403679988), intent=0 00000002:00010000:0.0:1788572596.701132:0:70938:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d133.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1788572596.701168:0:70938:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166a00/0x1e260726fbbc0ba1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572596.701346:0:70938:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166a00/0x1e260726fbbc0ba1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572596.701358:0:70938:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166a00/0x1e260726fbbc0ba1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572596.701474:0:70938:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70938:1875450488537472:192.168.202.125@tcp:101:lfs.0 00000100:00100000:1.0:1788572596.701576:0:70938:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408c8b900 going to sleep for 11 seconds 00000100:00100000:1.0:1788572596.709618:0:70938:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70938:1875450488537472:192.168.202.125@tcp:101:lfs.0 00010000:00010000:1.0:1788572596.709643:0:70938:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166a00/0x1e260726fbbc0ba1 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f64 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.709657:0:70938:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166a00/0x1e260726fbbc0ba1 lrc: 4/1,0 mode: --/PR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f64 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.709663:0:70938:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166a00/0x1e260726fbbc0ba1 lrc: 4/1,0 mode: --/PR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f64 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.709670:0:70938:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166a00/0x1e260726fbbc0ba1 lrc: 4/1,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0f64 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.709677:0:70938:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166a00/0x1e260726fbbc0ba1 lrc: 4/1,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f64 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.709704:0:70938:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166a00/0x1e260726fbbc0ba1 lrc: 5/2,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0f64 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.709713:0:70938:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166a00/0x1e260726fbbc0ba1 lrc: 5/2,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0f64 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.709720:0:70938:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166a00/0x1e260726fbbc0ba1 lrc: 4/1,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0f64 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572596.709733:0:70938:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff94d408d3ed80 x1875450488537472/t0(0) o101->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 592/584 e 0 to 0 dl 1788572612 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572596.709746:0:70938:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166a00/0x1e260726fbbc0ba1 lrc: 3/1,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f64 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.709757:0:70938:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d4420c9000 type 13 mode 4 res 9663703441/12 (0 0) 00000002:00010000:1.0:1788572596.709773:0:70938:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240006991:0xc:0x0], intent: lookup flags 0 00010000:00010000:1.0:1788572596.709819:0:70938:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583000/0x1e260726fbbc0ba8 lrc: 3/1,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.709827:0:70938:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583000/0x1e260726fbbc0ba8 lrc: 3/1,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.709835:0:70938:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583000/0x1e260726fbbc0ba8 lrc: 3/1,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572596.709854:0:70938:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70938:1875450488537600:192.168.202.125@tcp:101:lfs.0 00000100:00100000:1.0:1788572596.709924:0:70938:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408c8b900 going to sleep for 11 seconds 00000100:00100000:1.0:1788572596.720666:0:70938:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70938:1875450488537600:192.168.202.125@tcp:101:lfs.0 00010000:00010000:1.0:1788572596.720700:0:70938:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583000/0x1e260726fbbc0ba8 lrc: 4/1,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f72 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.720714:0:70938:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583000/0x1e260726fbbc0ba8 lrc: 4/1,0 mode: --/PR res: [0x240006991:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f72 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.720726:0:70938:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583000/0x1e260726fbbc0ba8 lrc: 4/1,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0f72 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.720738:0:70938:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583000/0x1e260726fbbc0ba8 lrc: 4/1,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f72 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.720750:0:70938:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583000/0x1e260726fbbc0ba8 lrc: 5/2,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0f72 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.720759:0:70938:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583000/0x1e260726fbbc0ba8 lrc: 5/2,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0f72 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.720766:0:70938:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583000/0x1e260726fbbc0ba8 lrc: 4/1,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0f72 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572596.720778:0:70938:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff94d4049bd880 x1875450488537600/t0(0) o101->lustre-MDT0001-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/584 e 0 to 0 dl 1788572612 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572596.720791:0:70938:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583000/0x1e260726fbbc0ba8 lrc: 3/1,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f72 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.720804:0:70938:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d404a46c00 type 13 mode 4 res 9663703441/12 (0 0) 00000080:00200000:1.0:1788572596.720824:0:70938:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572596.720846:0:70938:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x240006991:0xc:0x0], updating i_size 4096 00000080:00200000:1.0:1788572596.720850:0:70938:0:(namei.c:147:ll_iget()) got inode: [0x240006991:0xc:0x0](ffff94d403436c08): rc = 0 00000080:00010000:1.0:1788572596.720855:0:70938:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x240006991:0xc:0x0](ffff94d403436c08) for remote lock 0x1e260726fbbc0ba8 00000080:00010000:1.0:1788572596.720860:0:70938:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240006991:0xc:0x0](ffff94d403436c08) for lock 0x1e260726fbbc0ba1 00000080:00010000:1.0:1788572596.720870:0:70938:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240006991:0xc:0x0](ffff94d403436c08) 00000080:00010000:1.0:1788572596.720874:0:70938:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0ba1 from it ffffb68a41ff7a28 00010000:00010000:1.0:1788572596.720878:0:70938:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166a00/0x1e260726fbbc0ba1 lrc: 3/1,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0f64 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.720888:0:70938:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166a00/0x1e260726fbbc0ba1 lrc: 2/0,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f64 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1788572596.720899:0:70938:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x1e260726fbbc0ba8 from it ffffb68a41ff7a28 00010000:00010000:1.0:1788572596.720901:0:70938:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583000/0x1e260726fbbc0ba8 lrc: 3/1,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0f72 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.720907:0:70938:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583000/0x1e260726fbbc0ba8 lrc: 2/0,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f72 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572596.720924:0:70938:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x240006991:0xc:0x0](ffff94d403436c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572596.720942:0:70938:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=remote_dir, dir=[0x240006991:0xc:0x0](ffff94d403436c08), flags=13 00000080:00200000:1.0:1788572596.720946:0:70938:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=remote_dir, dir=[0x240006991:0xc:0x0](ffff94d403436c08), intent=getattr 00000002:00010000:1.0:1788572596.720966:0:70938:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: remote_dir,[0x0:0x0:0x0]) in obj [0x240006991:0xc:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788572596.721003:0:70938:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0baf lrc: 3/1,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.721010:0:70938:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0baf lrc: 3/1,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.721019:0:70938:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0baf lrc: 3/1,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572596.721038:0:70938:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70938:1875450488537728:192.168.202.125@tcp:101:lfs.0 00000100:00100000:1.0:1788572596.721121:0:70938:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408c8b900 going to sleep for 11 seconds 00000100:00100000:1.0:1788572596.723574:0:70938:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70938:1875450488537728:192.168.202.125@tcp:101:lfs.0 00010000:00010000:1.0:1788572596.723594:0:70938:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0baf lrc: 4/1,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.723606:0:70938:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0baf lrc: 4/1,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.723610:0:70938:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0baf lrc: 4/1,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.723614:0:70938:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0baf lrc: 3/0,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.723619:0:70938:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0baf lrc: 4/0,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.723623:0:70938:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412582c00 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:1.0:1788572596.723626:0:70938:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0baf lrc: 4/0,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.723632:0:70938:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0baf lrc: 4/0,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.723639:0:70938:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0baf lrc: 5/0,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.723646:0:70938:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572596.723652:0:70938:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0baf lrc: 3/0,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.723658:0:70938:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0baf lrc: 0/0,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572596.723675:0:70938:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff94d4049bd880 x1875450488537728/t0(0) o101->lustre-MDT0001-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 584/584 e 0 to 0 dl 1788572612 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572596.723699:0:70938:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240006991:0xc:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1788572596.723720:0:70938:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d404a46c00 type 13 mode 30 res 9663703441/12 (0 0) 00000080:00200000:1.0:1788572596.723780:0:70938:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572596.723793:0:70938:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572596.723802:0:70938:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572596.723811:0:70938:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572596.723815:0:70938:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572596.723819:0:70938:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572596.723821:0:70938:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0b9a 00000080:00010000:1.0:1788572596.723826:0:70938:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b9a from it ffffb68a41ff7be8 00010000:00010000:1.0:1788572596.723829:0:70938:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572596.723834:0:70938:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572596.723844:0:70938:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572596.723848:0:70938:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d133.replay-single, flags=16408 00000080:00200000:1.0:1788572596.723852:0:70938:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x240006991:0xc:0x0](ffff94d403436c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572596.723861:0:70938:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=remote_dir, dir=[0x240006991:0xc:0x0](ffff94d403436c08), flags=16392 00000080:00200000:1.0:1788572596.723864:0:70938:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=remote_dir, dir=[0x240006991:0xc:0x0](ffff94d403436c08), intent=getattr 00000002:00010000:1.0:1788572596.723872:0:70938:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: remote_dir,[0x0:0x0:0x0]) in obj [0x240006991:0xc:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788572596.723897:0:70938:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583c00/0x1e260726fbbc0bb6 lrc: 3/1,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.723902:0:70938:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583c00/0x1e260726fbbc0bb6 lrc: 3/1,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.723908:0:70938:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583c00/0x1e260726fbbc0bb6 lrc: 3/1,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572596.723920:0:70938:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70938:1875450488537856:192.168.202.125@tcp:101:lfs.0 00000100:00100000:1.0:1788572596.724011:0:70938:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408c8b900 going to sleep for 11 seconds 00000100:00100000:1.0:1788572596.726675:0:70938:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70938:1875450488537856:192.168.202.125@tcp:101:lfs.0 00010000:00010000:1.0:1788572596.726703:0:70938:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583c00/0x1e260726fbbc0bb6 lrc: 4/1,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.726716:0:70938:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583c00/0x1e260726fbbc0bb6 lrc: 4/1,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.726722:0:70938:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583c00/0x1e260726fbbc0bb6 lrc: 4/1,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.726728:0:70938:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583c00/0x1e260726fbbc0bb6 lrc: 3/0,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.726736:0:70938:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583c00/0x1e260726fbbc0bb6 lrc: 4/0,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.726742:0:70938:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412583c00 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:1.0:1788572596.726745:0:70938:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583c00/0x1e260726fbbc0bb6 lrc: 4/0,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.726752:0:70938:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583c00/0x1e260726fbbc0bb6 lrc: 4/0,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.726759:0:70938:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583c00/0x1e260726fbbc0bb6 lrc: 5/0,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.726765:0:70938:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572596.726771:0:70938:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583c00/0x1e260726fbbc0bb6 lrc: 3/0,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.726777:0:70938:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583c00/0x1e260726fbbc0bb6 lrc: 0/0,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572596.726793:0:70938:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff94d4049bd880 x1875450488537856/t0(0) o101->lustre-MDT0001-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 584/584 e 0 to 0 dl 1788572612 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572596.726806:0:70938:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240006991:0xc:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1788572596.726829:0:70938:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d404a46c00 type 13 mode 30 res 9663703441/12 (0 0) 00000080:00200000:1.0:1788572596.726882:0:70938:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572596.726892:0:70938:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572596.726898:0:70938:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572596.726909:0:70938:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572596.726913:0:70938:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572596.726916:0:70938:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572596.726918:0:70938:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0b9a 00000080:00010000:1.0:1788572596.726922:0:70938:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b9a from it ffffb68a41ff7b58 00010000:00010000:1.0:1788572596.726924:0:70938:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572596.726929:0:70938:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788572596.726939:0:70938:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572596.726945:0:70938:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d133.replay-single, flags=13 00000080:00200008:1.0:1788572596.726953:0:70938:0:(file.c:6285:ll_getattr_dentry()) START file d133.replay-single:[0x240006991:0xc:0x0](ffff94d403436c08), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1788572596.726958:0:70938:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x240006991:0xc:0x0](ffff94d403436c08),name=d133.replay-single 00000002:00010000:1.0:1788572596.726969:0:70938:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240006991:0xc:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572596.726974:0:70938:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d404a46c00 type 13 mode 30 res 9663703441/12 (0 0) 00010000:00010000:1.0:1788572596.727005:0:70938:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583200/0x1e260726fbbc0bbd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.727012:0:70938:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583200/0x1e260726fbbc0bbd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.727021:0:70938:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583200/0x1e260726fbbc0bbd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572596.727045:0:70938:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70938:1875450488537984:192.168.202.125@tcp:101:lfs.0 00000100:00100000:1.0:1788572596.727169:0:70938:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408c8b900 going to sleep for 11 seconds 00000100:00100000:1.0:1788572596.729071:0:70938:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70938:1875450488537984:192.168.202.125@tcp:101:lfs.0 00010000:00010000:1.0:1788572596.729101:0:70938:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583200/0x1e260726fbbc0bbd lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0faa expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.729121:0:70938:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583200/0x1e260726fbbc0bbd lrc: 4/1,0 mode: --/PR res: [0x240006991:0xc:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0faa expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.729128:0:70938:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583200/0x1e260726fbbc0bbd lrc: 4/1,0 mode: --/PR res: [0x240006991:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0faa expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.729138:0:70938:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583200/0x1e260726fbbc0bbd lrc: 4/1,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0faa expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.729147:0:70938:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583200/0x1e260726fbbc0bbd lrc: 4/1,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0faa expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.729158:0:70938:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583200/0x1e260726fbbc0bbd lrc: 5/2,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0faa expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.729165:0:70938:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583200/0x1e260726fbbc0bbd lrc: 5/2,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0faa expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.729171:0:70938:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583200/0x1e260726fbbc0bbd lrc: 4/1,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0faa expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572596.729557:0:70938:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d4049bd880 x1875450488537984/t0(0) o101->lustre-MDT0001-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 576/584 e 0 to 0 dl 1788572612 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572596.729570:0:70938:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583200/0x1e260726fbbc0bbd lrc: 3/1,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0faa expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.729580:0:70938:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d404a46c00 type 13 mode 4 res 9663703441/12 (0 0) 00000080:00200000:1.0:1788572596.729585:0:70938:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572596.729604:0:70938:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x240006991:0xc:0x0], updating i_size 4096 00000080:00010000:1.0:1788572596.729610:0:70938:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240006991:0xc:0x0](ffff94d403436c08) for lock 0x1e260726fbbc0bbd 00000080:00010000:1.0:1788572596.729615:0:70938:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240006991:0xc:0x0](ffff94d403436c08) 00000080:00010000:1.0:1788572596.729617:0:70938:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0bbd from it ffffb68a41ff7cb8 00010000:00010000:1.0:1788572596.729621:0:70938:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583200/0x1e260726fbbc0bbd lrc: 3/1,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0faa expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.729629:0:70938:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583200/0x1e260726fbbc0bbd lrc: 2/0,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0faa expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:1.0:1788572596.730343:0:70938:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1788572596.730349:0:70938:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572596.730419:0:70938:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572596.730431:0:70938:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572596.730442:0:70938:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788572596.730455:0:70938:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572596.730460:0:70938:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572596.730464:0:70938:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0b9a 00000080:00010000:1.0:1788572596.730468:0:70938:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b9a from it ffffb68a41ff7cb8 00010000:00010000:1.0:1788572596.730471:0:70938:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572596.730478:0:70938:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788572596.730602:0:70938:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:1.0:1788572596.730611:0:70938:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572596.730618:0:70938:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572596.730625:0:70938:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788572596.730631:0:70938:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572596.730635:0:70938:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:1.0:1788572596.730638:0:70938:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0b9a 00000080:00010000:1.0:1788572596.730642:0:70938:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b9a from it ffffb68a41ff7bf8 00010000:00010000:1.0:1788572596.730645:0:70938:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572596.730652:0:70938:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1788572596.730661:0:70938:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572596.730667:0:70938:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d133.replay-single, flags=267 00000080:00200000:1.0:1788572596.730672:0:70938:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x240006991:0xc:0x0](ffff94d403436c08), inode mode 41ed mask 44 00000080:00200008:1.0:1788572596.730676:0:70938:0:(file.c:1090:ll_file_open()) START file d133.replay-single:[0x240006991:0xc:0x0](ffff94d403436c08), flags 300000 00000002:00010000:1.0:1788572596.730698:0:70938:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240006991:0xc:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0240300001 00010000:00010000:1.0:1788572596.730732:0:70938:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0bc4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.730737:0:70938:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0bc4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572596.730746:0:70938:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572596.730753:0:70938:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d4049bd880 x1875450488538240/t0(0) o101->lustre-MDT0001-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572596.730761:0:70938:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0bc4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572596.730781:0:70938:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70938:1875450488538240:192.168.202.125@tcp:101:lfs.0 00000100:00100000:1.0:1788572596.730936:0:70938:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408c8b900 going to sleep for 11 seconds 00000100:00100000:1.0:1788572596.733053:0:70938:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70938:1875450488538240:192.168.202.125@tcp:101:lfs.0 00010000:00010000:1.0:1788572596.733074:0:70938:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0bc4 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.733091:0:70938:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0bc4 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.733097:0:70938:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0bc4 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.733104:0:70938:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0bc4 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.733111:0:70938:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0bc4 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.733117:0:70938:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412582c00 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:1.0:1788572596.733121:0:70938:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0bc4 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.733127:0:70938:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0bc4 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.733133:0:70938:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0bc4 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.733138:0:70938:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572596.733143:0:70938:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0bc4 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572596.733149:0:70938:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412582c00/0x1e260726fbbc0bc4 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 70938 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572596.733170:0:70938:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff94d4049bd880 x1875450488538240/t81604378735(81604378735) o101->lustre-MDT0001-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572612 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572596.733182:0:70938:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d4049bd880 x1875450488538240/t81604378735(81604378735) o101->lustre-MDT0001-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572612 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572596.733194:0:70938:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572596.733259:0:70938:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x240006991:0xc:0x0], updating i_size 4096 00000002:00100000:1.0:1788572596.733276:0:70938:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff94d4049bd880 x1875450488538240/t81604378735(81604378735) o101->lustre-MDT0001-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572612 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:1.0:1788572596.733354:0:70938:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x240006991:0xc:0x0](ffff94d403436c08) cmd=c03066f0 arg=7ffecbd74eb0 00000080:00200000:1.0:1788572596.733368:0:70938:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x240006991:0xc:0x0](ffff94d403436c08) name=remote_dir stripe_offset=1 stripe_count=0, hash_type=0 00010000:00010000:1.0:1788572596.733394:0:70938:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583200/0x1e260726fbbc0bbd lrc: 2/0,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4e0faa expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572596.733413:0:70938:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583000/0x1e260726fbbc0ba8 lrc: 2/0,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x10/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f72 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572596.733447:0:70938:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583200/0x1e260726fbbc0bbd lrc: 1/0,0 mode: --/PR res: [0x240006991:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4e0faa expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572596.733454:0:70938:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572596.733468:0:70938:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583200/0x1e260726fbbc0bbd lrc: 0/0,0 mode: --/PR res: [0x240006991:0xc:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4e0faa expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1788572596.733483:0:70938:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788572596.733490:0:70938:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d4049bdc00 x1875450488538496/t0(0) o36->lustre-MDT0001-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572596.733510:0:70938:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70938:1875450488538496:192.168.202.125@tcp:36:lfs.0 00000100:00100000:1.0:1788572596.733966:0:70938:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408c8b900 going to sleep for 11 seconds 00000100:00100000:1.0:1788572596.736333:0:70938:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70938:1875450488538496:192.168.202.125@tcp:36:lfs.0 00000080:00200000:1.0:1788572596.736385:0:70938:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x240006991:0xd:0x0](ffff94d410bc1988) 00000080:00200000:1.0:1788572596.736391:0:70938:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x240006991:0xd:0x0], updating i_size 4096 00000080:00200008:1.0:1788572596.736426:0:70938:0:(file.c:479:ll_file_release()) START file d133.replay-single:[0x240006991:0xc:0x0](ffff94d403436c08), flags 300000 00000002:00100000:1.0:1788572596.736489:0:70938:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff94d4049bd880 x1875450488538240/t81604378735(81604378735) o101->lustre-MDT0001-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572612 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572596.736502:0:70938:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff94d405cc3800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572596.736508:0:70938:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff94d4049be300 x1875450488538752/t0(0) o35->lustre-MDT0001-mdc-ffff94d405cc3800@192.168.202.125@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572596.736532:0:70938:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70938:1875450488538752:192.168.202.125@tcp:35:lfs.0 00000100:00100000:1.0:1788572596.736596:0:70938:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408c8b900 going to sleep for 11 seconds 00000100:00100000:1.0:1788572596.738024:0:70938:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job lfs:9bf287c0-f582-4b83-84c1-15e766218620:70938:1875450488538752:192.168.202.125@tcp:35:lfs.0 00000002:00100000:1.0:1788572596.738047:0:70938:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff94d4049bd880 x1875450488538240/t81604378735(81604378735) o101->lustre-MDT0001-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572612 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788572596.744998:0:70939:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572596.745011:0:70939:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572596.745036:0:70939:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572596.745054:0:70939:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788572596.745068:0:70939:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1788572596.745074:0:70939:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572596.745081:0:70939:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572596.745084:0:70939:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0b9a 00000080:00010000:2.0:1788572596.745092:0:70939:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b9a from it ffffb68a40d1fcb8 00010000:00010000:2.0:1788572596.745095:0:70939:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788572596.745102:0:70939:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200008:2.0:1788572596.745235:0:70939:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d403679988), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572596.745240:0:70939:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988),name=/ 00000002:00010000:2.0:1788572596.745248:0:70939:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572596.745255:0:70939:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788572596.745264:0:70939:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1788572596.745270:0:70939:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572596.745275:0:70939:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) 00000080:00010000:2.0:1788572596.745278:0:70939:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d403679988) for lock 0x1e260726fbbc0b9a 00000080:00010000:2.0:1788572596.745281:0:70939:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0b9a from it ffffb68a40d1fcb8 00010000:00010000:2.0:1788572596.745285:0:70939:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788572596.745292:0:70939:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1788572596.754502:0:51031:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572596.754521:0:51031:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:1.0:1788572596.754545:0:51031:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:1.0:1788572596.754646:0:51031:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572596.754778:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d403679988) 00000080:00200000:2.0:1788572596.754810:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000059f2:0x1:0x0](ffff94d410221988) 00000080:00200000:2.0:1788572596.754818:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c1:0x1:0x0](ffff94d403682a08) 00000080:00200000:2.0:1788572596.754825:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x240006990:0x3:0x0](ffff94d410224b08) 00000080:00200000:2.0:1788572596.754831:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c2:0x1:0x0](ffff94d410225348) 00000080:00200000:2.0:1788572596.754837:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c1:0x4:0x0](ffff94d403683a88) 00000080:00200000:2.0:1788572596.754843:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x240007160:0x3:0x0](ffff94d403695b88) 00000080:00200000:2.0:1788572596.754849:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200006991:0x1:0x0](ffff94d4036921c8) 00000080:00200000:2.0:1788572596.754855:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c1:0x7:0x0](ffff94d410227448) 00000080:00200000:2.0:1788572596.754860:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x240007930:0x1:0x0](ffff94d403691988) 00000080:00200000:2.0:1788572596.754867:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200007161:0x3:0x0](ffff94d4036900c8) 00000080:00200000:2.0:1788572596.754874:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x240006991:0x4:0x0](ffff94d403694b08) 00000080:00200000:2.0:1788572596.754879:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x240008100:0x3:0x0](ffff94d40367aa08) 00000080:00200000:2.0:1788572596.754885:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200007931:0x3:0x0](ffff94d40367a1c8) 00000080:00200000:2.0:1788572596.754890:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x240008100:0x4:0x0](ffff94d4102200c8) 00000080:00200000:2.0:1788572596.754897:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200007931:0x4:0x0](ffff94d410223248) 00000080:00200000:2.0:1788572596.754903:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x240008100:0x5:0x0](ffff94d4036963c8) 00000080:00200000:2.0:1788572596.754909:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200007931:0x5:0x0](ffff94d403695348) 00000080:00200000:2.0:1788572596.754914:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x240008100:0x6:0x0](ffff94d40367e3c8) 00000080:00200000:2.0:1788572596.754922:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200007931:0x6:0x0](ffff94d40367c2c8) 00000080:00200000:2.0:1788572596.754931:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x240008100:0x7:0x0](ffff94d4036800c8) 00000080:00200000:2.0:1788572596.754941:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200007931:0x7:0x0](ffff94d403681988) 00000080:00200000:2.0:1788572596.754950:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200007930:0x1:0x0](ffff94d4102263c8) 00000080:00200000:2.0:1788572596.754958:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400088d1:0x1:0x0](ffff94d410222a08) 00000080:00200000:2.0:1788572596.754967:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200007930:0x4:0x0](ffff94d4036942c8) 00000080:00200000:2.0:1788572596.754972:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400088d1:0x4:0x0](ffff94d40367cb08) 00000080:00200000:2.0:1788572596.754978:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200007930:0x5:0x0](ffff94d403678908) 00000080:00200000:2.0:1788572596.754983:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400088d1:0x5:0x0](ffff94d40367b248) 00000080:00200000:2.0:1788572596.754988:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200007930:0x6:0x0](ffff94d4102221c8) 00000080:00200000:2.0:1788572596.754993:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400088d1:0x6:0x0](ffff94d410221148) 00000080:00200000:2.0:1788572596.754999:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200007930:0x7:0x0](ffff94d410223a88) 00000080:00200000:2.0:1788572596.755004:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400088d1:0x7:0x0](ffff94d410226c08) 00000080:00200000:2.0:1788572596.755009:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c1:0x11:0x0](ffff94d410225b88) 00000080:00200000:2.0:1788572596.755016:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c1:0x13:0x0](ffff94d410220908) 00000080:00200000:2.0:1788572596.755020:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200008100:0x3:0x0](ffff94d403691148) 00000080:00200000:2.0:1788572596.755026:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400088d2:0x3:0x0](ffff94d403693248) 00000080:00200000:2.0:1788572596.755033:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c1:0x14:0x0](ffff94d403690908) 00000080:00200000:2.0:1788572596.755039:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c1:0x16:0x0](ffff94d403681148) 00000080:00200000:2.0:1788572596.755043:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000088d0:0x3:0x0](ffff94d403692a08) 00000080:00200000:2.0:1788572596.755048:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400088d3:0x3:0x0](ffff94d403697448) 00000080:00200000:2.0:1788572596.755056:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c1:0x17:0x0](ffff94d4102242c8) 00000080:00200000:2.0:1788572596.755061:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000088d0:0x4:0x0](ffff94d403696c08) 00000080:00200000:2.0:1788572596.755066:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400088d3:0x4:0x0](ffff94d403693a88) 00000080:00200000:2.0:1788572596.755071:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000088d0:0x5:0x0](ffff94d4109e5348) 00000080:00200000:2.0:1788572596.755076:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400088d3:0x5:0x0](ffff94d4109e63c8) 00000080:00200000:2.0:1788572596.755081:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x240006991:0xb:0x0](ffff94d403683248) 00000080:00200000:2.0:1788572596.755088:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c1:0x1c:0x0](ffff94d410bd42c8) 00000080:00200000:2.0:1788572596.755094:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c1:0x1d:0x0](ffff94d410bd3a88) 00000080:00200000:2.0:1788572596.755101:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c1:0x20:0x0](ffff94d4109e2a08) 00000080:00200000:2.0:1788572596.755108:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c1:0x21:0x0](ffff94d4109e21c8) 00000080:00200000:2.0:1788572596.755111:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000090a0:0x3:0x0](ffff94d4109e5b88) 00000080:00200000:2.0:1788572596.755118:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a1:0x3:0x0](ffff94d4034caa08) 00000080:00200000:2.0:1788572596.755126:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c1:0x22:0x0](ffff94d4109e4b08) 00000080:00200000:2.0:1788572596.755130:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000090a0:0x4:0x0](ffff94d4034cec08) 00000080:00200000:2.0:1788572596.755136:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a1:0x4:0x0](ffff94d4034cd348) 00000080:00200000:2.0:1788572596.755145:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c1:0x23:0x0](ffff94d40367ba88) 00000080:00200000:2.0:1788572596.755149:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000090a0:0x5:0x0](ffff94d403680908) 00000080:00200000:2.0:1788572596.755154:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a1:0x5:0x0](ffff94d4036842c8) 00000080:00200000:2.0:1788572596.755163:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c1:0x24:0x0](ffff94d4109e7448) 00000080:00200000:2.0:1788572596.755168:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000090a0:0x6:0x0](ffff94d410bd5b88) 00000080:00200000:2.0:1788572596.755176:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a1:0x6:0x0](ffff94d410bd63c8) 00000080:00200000:2.0:1788572596.755189:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c1:0x25:0x0](ffff94d410bd6c08) 00000080:00200000:2.0:1788572596.755196:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000090a0:0x7:0x0](ffff94d4034c9988) 00000080:00200000:2.0:1788572596.755238:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a1:0x7:0x0](ffff94d4034c9148) 00000080:00200000:2.0:1788572596.755253:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c1:0x26:0x0](ffff94d4109e42c8) 00000080:00200000:2.0:1788572596.755259:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000090a0:0x8:0x0](ffff94d4034cf448) 00000080:00200000:2.0:1788572596.755267:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a1:0x8:0x0](ffff94d4034cdb88) 00000080:00200000:2.0:1788572596.755277:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c1:0x27:0x0](ffff94d4109e3a88) 00000080:00200000:2.0:1788572596.755280:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000090a0:0x9:0x0](ffff94d4034c80c8) 00000080:00200000:2.0:1788572596.755286:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a1:0x9:0x0](ffff94d4034cc2c8) 00000080:00200000:2.0:1788572596.755296:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c1:0x28:0x0](ffff94d410bd1148) 00000080:00200000:2.0:1788572596.755300:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000090a0:0xa:0x0](ffff94d410bd21c8) 00000080:00200000:2.0:1788572596.755305:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a1:0xa:0x0](ffff94d410bd4b08) 00000080:00200000:2.0:1788572596.755313:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c1:0x29:0x0](ffff94d410bd1988) 00000080:00200000:2.0:1788572596.755318:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000090a0:0xb:0x0](ffff94d40367db88) 00000080:00200000:2.0:1788572596.755323:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a1:0xb:0x0](ffff94d40367d348) 00000080:00200000:2.0:1788572596.755331:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c1:0x2a:0x0](ffff94d4109e00c8) 00000080:00200000:2.0:1788572596.755335:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000090a0:0xc:0x0](ffff94d410bd7448) 00000080:00200000:2.0:1788572596.755340:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a1:0xc:0x0](ffff94d410bd00c8) 00000080:00200000:2.0:1788572596.755349:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c1:0x2b:0x0](ffff94d410bd3248) 00000080:00200000:2.0:1788572596.755353:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000090a0:0xd:0x0](ffff94d410bcdb88) 00000080:00200000:2.0:1788572596.755359:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a1:0xd:0x0](ffff94d410bce3c8) 00000080:00200000:2.0:1788572596.755367:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c1:0x2c:0x0](ffff94d4109e6c08) 00000080:00200000:2.0:1788572596.755371:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000090a0:0xe:0x0](ffff94d403679148) 00000080:00200000:2.0:1788572596.755377:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a1:0xe:0x0](ffff94d40367ec08) 00000080:00200000:2.0:1788572596.755385:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c1:0x2d:0x0](ffff94d4109e1148) 00000080:00200000:2.0:1788572596.755389:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000090a0:0xf:0x0](ffff94d410bca1c8) 00000080:00200000:2.0:1788572596.755403:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a1:0xf:0x0](ffff94d410bccb08) 00000080:00200000:2.0:1788572596.755417:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c1:0x2e:0x0](ffff94d410bd5348) 00000080:00200000:2.0:1788572596.755423:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000090a0:0x10:0x0](ffff94d410bcf448) 00000080:00200000:2.0:1788572596.755432:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a1:0x10:0x0](ffff94d410bcc2c8) 00000080:00200000:2.0:1788572596.755444:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c1:0x2f:0x0](ffff94d4109e1988) 00000080:00200000:2.0:1788572596.755449:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000090a0:0x11:0x0](ffff94d4036780c8) 00000080:00200000:2.0:1788572596.755454:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a1:0x11:0x0](ffff94d40367f448) 00000080:00200000:2.0:1788572596.755470:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c1:0x30:0x0](ffff94d410bd0908) 00000080:00200000:2.0:1788572596.755474:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000090a0:0x12:0x0](ffff94d410bcba88) 00000080:00200000:2.0:1788572596.755480:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a1:0x12:0x0](ffff94d410bc80c8) 00000080:00200000:2.0:1788572596.755488:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c1:0x31:0x0](ffff94d403686c08) 00000080:00200000:2.0:1788572596.755491:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000090a0:0x13:0x0](ffff94d4034ccb08) 00000080:00200000:2.0:1788572596.755497:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a1:0x13:0x0](ffff94d4034c8908) 00000080:00200000:2.0:1788572596.755506:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c1:0x32:0x0](ffff94d4109e3248) 00000080:00200000:2.0:1788572596.755509:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000090a0:0x14:0x0](ffff94d4034ca1c8) 00000080:00200000:2.0:1788572596.755515:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a1:0x14:0x0](ffff94d4034cb248) 00000080:00200000:2.0:1788572596.755523:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c1:0x33:0x0](ffff94d410bd2a08) 00000080:00200000:2.0:1788572596.755526:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000090a0:0x15:0x0](ffff94d410bcec08) 00000080:00200000:2.0:1788572596.755532:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a1:0x15:0x0](ffff94d410bc9148) 00000080:00200000:2.0:1788572596.755540:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000061c1:0x34:0x0](ffff94d4109e0908) 00000080:00200000:2.0:1788572596.755543:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2000090a0:0x16:0x0](ffff94d4034ce3c8) 00000080:00200000:2.0:1788572596.755548:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a1:0x16:0x0](ffff94d4034cba88) 00000080:00200000:2.0:1788572596.755556:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200009870:0x4:0x0](ffff94d40353d348) 00000080:00200000:2.0:1788572596.755561:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a2:0x2:0x0](ffff94d40353db88) 00000080:00200000:2.0:1788572596.755566:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200009870:0x7:0x0](ffff94d410bc4b08) 00000080:00200000:2.0:1788572596.755571:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a2:0x5:0x0](ffff94d410bc21c8) 00000080:00200000:2.0:1788572596.755582:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200009870:0x10:0x0](ffff94d410bc2a08) 00000080:00200000:2.0:1788572596.755592:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a2:0xe:0x0](ffff94d410bc00c8) 00000080:00200000:2.0:1788572596.755601:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200009870:0x11:0x0](ffff94d40353ba88) 00000080:00200000:2.0:1788572596.755610:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a2:0xf:0x0](ffff94d40353e3c8) 00000080:00200000:2.0:1788572596.755619:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200009870:0x12:0x0](ffff94d410bc6c08) 00000080:00200000:2.0:1788572596.755627:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a2:0x10:0x0](ffff94d410bc5348) 00000080:00200000:2.0:1788572596.755635:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200009870:0x13:0x0](ffff94d410bc0908) 00000080:00200000:2.0:1788572596.755644:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a2:0x11:0x0](ffff94d410bc42c8) 00000080:00200000:2.0:1788572596.755652:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200009870:0x14:0x0](ffff94d410bc5b88) 00000080:00200000:2.0:1788572596.755661:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a2:0x12:0x0](ffff94d403495348) 00000080:00200000:2.0:1788572596.755669:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200009870:0x15:0x0](ffff94d403493a88) 00000080:00200000:2.0:1788572596.755677:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a2:0x13:0x0](ffff94d403491148) 00000080:00200000:2.0:1788572596.755697:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200009870:0x16:0x0](ffff94d4034921c8) 00000080:00200000:2.0:1788572596.755706:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a2:0x14:0x0](ffff94d403497448) 00000080:00200000:2.0:1788572596.755714:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200009870:0x17:0x0](ffff94d403492a08) 00000080:00200000:2.0:1788572596.755722:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a2:0x15:0x0](ffff94d403490908) 00000080:00200000:2.0:1788572596.755729:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200009870:0x18:0x0](ffff94d403494b08) 00000080:00200000:2.0:1788572596.755736:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a2:0x16:0x0](ffff94d4034942c8) 00000080:00200000:2.0:1788572596.755744:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200009870:0x19:0x0](ffff94d403539148) 00000080:00200000:2.0:1788572596.755751:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a2:0x17:0x0](ffff94d40353ec08) 00000080:00200000:2.0:1788572596.755760:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200009870:0x8:0x0](ffff94d4035380c8) 00000080:00200000:2.0:1788572596.755767:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a2:0x6:0x0](ffff94d403539988) 00000080:00200000:2.0:1788572596.755774:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200009870:0x9:0x0](ffff94d403538908) 00000080:00200000:2.0:1788572596.755781:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a2:0x7:0x0](ffff94d40353b248) 00000080:00200000:2.0:1788572596.755788:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200009870:0xa:0x0](ffff94d40353cb08) 00000080:00200000:2.0:1788572596.755797:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a2:0x8:0x0](ffff94d40353c2c8) 00000080:00200000:2.0:1788572596.755805:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200009870:0xb:0x0](ffff94d40353aa08) 00000080:00200000:2.0:1788572596.755814:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a2:0x9:0x0](ffff94d40353a1c8) 00000080:00200000:2.0:1788572596.755820:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200009870:0xc:0x0](ffff94d403495b88) 00000080:00200000:2.0:1788572596.755826:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a2:0xa:0x0](ffff94d403496c08) 00000080:00200000:2.0:1788572596.755832:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200009870:0xd:0x0](ffff94d40353f448) 00000080:00200000:2.0:1788572596.755837:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a2:0xb:0x0](ffff94d410bcd348) 00000080:00200000:2.0:1788572596.755842:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200009870:0xe:0x0](ffff94d403685b88) 00000080:00200000:2.0:1788572596.755848:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a2:0xc:0x0](ffff94d403684b08) 00000080:00200000:2.0:1788572596.755853:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200009870:0xf:0x0](ffff94d403491988) 00000080:00200000:2.0:1788572596.755861:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x2400090a2:0xd:0x0](ffff94d403493248) 00000080:00200000:2.0:1788572596.755890:0:70939:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572596.755896:0:70939:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:2.0:1788572596.755919:0:70939:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x0 00000080:00200000:2.0:1788572596.755945:0:70939:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572596.755959:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x20000a811:0x2:0x0](ffff94d410bcaa08) 00000080:00200000:2.0:1788572596.756019:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x20000a811:0x3:0x0](ffff94d4034300c8) 00000080:00200000:2.0:1788572596.756036:0:70939:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572596.756041:0:70939:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:2.0:1788572596.756048:0:70939:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x0 00000080:00200000:2.0:1788572596.756057:0:70939:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572596.756062:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x20000a811:0x5:0x0](ffff94d4034900c8) 00000080:00200000:2.0:1788572596.756091:0:70939:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572596.756094:0:70939:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:2.0:1788572596.756100:0:70939:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x0 00000080:00200000:2.0:1788572596.756106:0:70939:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572596.756111:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x20000a811:0x6:0x0](ffff94d4034963c8) 00000080:00200000:2.0:1788572596.756165:0:70939:0:(vvp_io.c:1826:vvp_io_init()) [0x20000a811:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572596.756169:0:70939:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:2.0:1788572596.756179:0:70939:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:2.0:1788572596.756271:0:70939:0:(vvp_io.c:304:vvp_io_fini()) [0x20000a811:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572596.758291:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x20000a811:0x9:0x0](ffff94d410bc3248) 00000008:00010000:2.0:1788572596.758344:0:70939:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 0 [cma]time: 0 0 1788572535 by oinfo size 0 blocks 0 [cma]time 0 0 1788572590 ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 1/0,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 9 lru_type: 1 00000008:00010000:2.0:1788572596.758366:0:70939:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 2097152 blocks 2048 [cma]time: 1788572535 1788572535 1788572535 by oinfo size 2097152 blocks 2048 [cma]time 1788572535 1788572535 1788572590 ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 1/0,0 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 10 lru_type: 1 00000008:00010000:2.0:1788572596.758407:0:70939:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 0 [cma]time: 1788572535 1788572535 0 by oinfo size 0 blocks 0 [cma]time 1788572535 1788572535 0 ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 1/0,0 mode: PW/PW res: [0x280000401:0x703:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xec901abfeb4e0a08 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00000080:00200000:2.0:1788572596.758455:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x240006991:0xc:0x0](ffff94d403436c08) 00000080:00200000:2.0:1788572596.758467:0:70939:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x240006991:0xd:0x0](ffff94d410bc1988) 00000080:00200000:2.0:1788572596.758475:0:70939:0:(llite_lib.c:1697:ll_put_super()) VFS Op: cfg_instance lustre-client-ffff94d405cc3800 (sb ffff94d405cc3800) 00010000:00010000:2.0:1788572596.758506:0:70939:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412166600/0x1e260726fbbc0b62 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0aa9 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572596.758514:0:70939:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412166600/0x1e260726fbbc0b62 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xec901abfeb4e0aa9 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572596.758520:0:70939:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412166600/0x1e260726fbbc0b62 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0xec901abfeb4e0aa9 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572596.758524:0:70939:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.125@tcp lock: ffff94d412166600/0x1e260726fbbc0b62 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0xec901abfeb4e0aa9 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572596.758541:0:70939:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:2.0:1788572596.758546:0:70939:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412582200/0x1e260726fbbc0b77 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0bcf expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572596.758549:0:70939:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412582200/0x1e260726fbbc0b77 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xec901abfeb4e0bcf expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572596.758553:0:70939:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412582200/0x1e260726fbbc0b77 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xec901abfeb4e0bcf expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572596.758558:0:70939:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.125@tcp lock: ffff94d412582200/0x1e260726fbbc0b77 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xec901abfeb4e0bcf expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572596.758583:0:70939:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to regular blp list, 2 blwis in pool 00010000:00010000:3.0:1788572596.758587:0:42449:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:2.0:1788572596.758590:0:70939:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412167000/0x1e260726fbbc0b69 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0ab0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572596.758600:0:42449:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.125@tcp lock: ffff94d412166600/0x1e260726fbbc0b62 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0xec901abfeb4e0aa9 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572596.758601:0:70939:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412167000/0x1e260726fbbc0b69 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xec901abfeb4e0ab0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572596.758618:0:42449:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412166600 already unused, calling callback (ffffffffc115cc20) 00010000:00010000:0.0:1788572596.758625:0:63655:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 10000000:00010000:3.0:1788572596.758627:0:42449:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.125@tcp lock: ffff94d412166600/0x1e260726fbbc0b62 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0xec901abfeb4e0aa9 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572596.758628:0:70939:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412167000/0x1e260726fbbc0b69 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xec901abfeb4e0ab0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572596.758634:0:63655:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.125@tcp lock: ffff94d412582200/0x1e260726fbbc0b77 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xec901abfeb4e0bcf expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572596.758649:0:70939:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.125@tcp lock: ffff94d412167000/0x1e260726fbbc0b69 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xec901abfeb4e0ab0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572596.758657:0:63655:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412582200 already unused, calling callback (ffffffffc115cc20) 10000000:00010000:0.0:1788572596.758663:0:63655:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.125@tcp lock: ffff94d412582200/0x1e260726fbbc0b77 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xec901abfeb4e0bcf expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572596.758670:0:42449:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.125@tcp lock: ffff94d412166600/0x1e260726fbbc0b62 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1008400000000 nid: local remote: 0xec901abfeb4e0aa9 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572596.758675:0:63655:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.125@tcp lock: ffff94d412582200/0x1e260726fbbc0b77 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1008400000000 nid: local remote: 0xec901abfeb4e0bcf expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572596.758693:0:70939:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 10000000:00010000:3.0:1788572596.758696:0:42449:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.125@tcp lock: ffff94d412166600/0x1e260726fbbc0b62 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1009400000000 nid: local remote: 0xec901abfeb4e0aa9 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788572596.758701:0:63655:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.125@tcp lock: ffff94d412582200/0x1e260726fbbc0b77 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1009400000000 nid: local remote: 0xec901abfeb4e0bcf expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572596.758701:0:70939:0:(mgc_request.c:547:config_log_end()) end config log lustre-client (0) 10000000:01000000:3.0:1788572596.758711:0:42449:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:0.0:1788572596.758714:0:63655:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 10000000:01000000:3.0:1788572596.758716:0:42449:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-client: stopping, won't requeue 10000000:01000000:0.0:1788572596.758719:0:63655:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-cliir: stopping, won't requeue 10000000:01000000:3.0:1788572596.758721:0:42449:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-client 10000000:01000000:0.0:1788572596.758723:0:63655:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-cliir 00010000:00010000:1.0:1788572596.758728:0:42448:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788572596.758773:0:63655:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.125@tcp lock: ffff94d412582200/0x1e260726fbbc0b77 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xec901abfeb4e0bcf expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572596.758773:0:42449:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.125@tcp lock: ffff94d412166600/0x1e260726fbbc0b62 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xec901abfeb4e0aa9 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572596.758790:0:42449:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572596.758792:0:63655:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572596.758826:0:63655:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.125@tcp lock: ffff94d412582200/0x1e260726fbbc0b77 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xec901abfeb4e0bcf expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572596.758828:0:42449:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.125@tcp lock: ffff94d412166600/0x1e260726fbbc0b62 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xec901abfeb4e0aa9 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572596.758845:0:63655:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.125@tcp lock: ffff94d412582200/0x1e260726fbbc0b77 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xec901abfeb4e0bcf expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572596.758847:0:42449:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.125@tcp lock: ffff94d412166600/0x1e260726fbbc0b62 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xec901abfeb4e0aa9 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572596.758852:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00010000:00010000:3.0:1788572596.758875:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572596.758879:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572596.758881:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572596.758883:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488539008:192.168.202.125@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1788572596.758884:0:42449:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572596.758886:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572596.758892:0:63655:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572596.758905:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488538880:192.168.202.125@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1788572596.759126:0:70940:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572596.759129:0:70940:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572596.759132:0:70940:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572596.759158:0:42448:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.125@tcp lock: ffff94d412167000/0x1e260726fbbc0b69 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xec901abfeb4e0ab0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572596.759170:0:42448:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412167000 already unused, calling callback (ffffffffc115cc20) 10000000:00010000:0.0:1788572596.759174:0:42448:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.125@tcp lock: ffff94d412167000/0x1e260726fbbc0b69 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xec901abfeb4e0ab0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572596.759183:0:42448:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.125@tcp lock: ffff94d412167000/0x1e260726fbbc0b69 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1008400000000 nid: local remote: 0xec901abfeb4e0ab0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788572596.759191:0:42448:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.125@tcp lock: ffff94d412167000/0x1e260726fbbc0b69 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1009400000000 nid: local remote: 0xec901abfeb4e0ab0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788572596.759198:0:42448:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 10000000:01000000:0.0:1788572596.759226:0:42448:0:(mgc_request.c:944:mgc_blocking_ast()) log params: stopping, won't requeue 10000000:01000000:0.0:1788572596.759229:0:42448:0:(mgc_request.c:130:config_log_put()) dropping config log params 00010000:00010000:0.0:1788572596.759251:0:42448:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.125@tcp lock: ffff94d412167000/0x1e260726fbbc0b69 lrc: 2/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xec901abfeb4e0ab0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572596.759258:0:42448:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572596.759290:0:42448:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.125@tcp lock: ffff94d412167000/0x1e260726fbbc0b69 lrc: 1/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xec901abfeb4e0ab0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572596.759298:0:42448:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.125@tcp lock: ffff94d412167000/0x1e260726fbbc0b69 lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xec901abfeb4e0ab0 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572596.759317:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572596.759321:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572596.759325:0:42448:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572596.759340:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488539136:192.168.202.125@tcp:103:ldlm_bl.0 00020000:01000000:2.0:1788572596.759634:0:70939:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0000_UUID idx: 0 gen: 1 exp: ffff94d4182fa000 active: 1 00020000:01000000:2.0:1788572596.759640:0:70939:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0001_UUID idx: 1 gen: 1 exp: ffff94d4182f8000 active: 1 00020000:01000000:2.0:1788572596.759643:0:70939:0:(lov_obd.c:64:lov_tgts_putref()) destroying 2 lov targets 00020000:01000000:2.0:1788572596.759648:0:70939:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0001_UUID : lustre-OST0001-osc-ffff94d405cc3800 00020000:01000000:2.0:1788572596.759651:0:70939:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff94d405cc3800: disconnecting target lustre-OST0001-osc-ffff94d405cc3800 00000100:00080000:2.0:1788572596.759662:0:70939:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 9bf287c0-f582-4b83-84c1-15e766218620->lustre-OST0001_UUID 00010000:00010000:2.0:1788572596.759692:0:70939:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x28400000000 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 10 lru_type: 1 00010000:00010000:2.0:1788572596.759733:0:70939:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 1/0,0 mode: --/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x4829400000020 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 10 lru_type: 1 00010000:00010000:2.0:1788572596.759741:0:70939:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:2.0:1788572596.759751:0:70939:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408529900 going to sleep for 0 seconds 00000100:00100000:2.0:1788572596.759758:0:70939:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job umount:9bf287c0-f582-4b83-84c1-15e766218620:70939:1875450488539264:192.168.202.125@tcp:103:umount.0 00010000:00010000:2.0:1788572596.759765:0:70939:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:2.0:1788572596.759773:0:70939:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:2.0:1788572596.759776:0:70939:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff94d405cc3800 lock: 0000000061095bae/0x1e260726fbbc0b38 lrc: 0/0,0 mode: --/PW res: [0x2c0000401:0x702:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x4829400000020 nid: local remote: 0xec901abfeb4e0a01 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 10 lru_type: 1 00000100:00080000:2.0:1788572596.759884:0:70939:0:(import.c:65:import_set_state_nolock()) ffff94d405bab800 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:00100000:3.0:1788572596.759957:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488539392:192.168.202.125@tcp:9:umount.0 00000100:00100000:0.0:1788572596.760999:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488539008:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572596.761023:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2404:1875450488539136:192.168.202.125@tcp:103:ldlm_bl.0 00000100:00080000:3.0:1788572596.761562:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab800 lustre-OST0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:3.0:1788572596.761596:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488539392:192.168.202.125@tcp:9:umount.0 00000100:00080000:2.0:1788572596.761700:0:70939:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:2.0:1788572596.761705:0:70939:0:(client.c:3112:ptlrpc_free_committed()) lustre-OST0001-osc-ffff94d405cc3800: committing for last_committed 4294968863 gen 18 00000100:00100000:2.0:1788572596.761712:0:70939:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572596.761899:0:70939:0:(genops.c:1489:class_disconnect()) disconnect: cookie 0x1e260726fbbbf4fd 00000020:00000080:2.0:1788572596.761920:0:70939:0:(genops.c:906:class_export_put()) final put ffff94d4182f8000/LOV_OSC_UUID 00000100:00100000:3.0:1788572596.761929:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e60431de-58b9-4f69-9048-11a3d82ddcdd:2406:1875450488538880:192.168.202.125@tcp:103:ldlm_bl.0 00000020:01000000:2.0:1788572596.761968:0:70939:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0001-osc-ffff94d405cc3800 (flags='') 00000020:00000080:2.0:1788572596.761973:0:70939:0:(obd_config.c:1470:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572596.761975:0:51031:0:(genops.c:865:class_export_destroy()) destroying export ffff94d4182f8000/LOV_OSC_UUID for lustre-OST0001-osc-ffff94d405cc3800 00000020:00000080:2.0:1788572596.761984:0:70939:0:(obd_config.c:955:class_cleanup()) lustre-OST0001-osc-ffff94d405cc3800: forcing exports to disconnect: 0/2 00000020:00080000:2.0:1788572596.761989:0:70939:0:(genops.c:1579:class_disconnect_exports()) OBD device 6 (ffff94d407dbb380) has no exports 00000008:01000000:2.0:1788572596.762898:0:70939:0:(obd_class.h:671:obd_cleanup_client_import()) lustre-OST0001-osc-ffff94d405cc3800: client import never connected 00000100:00080000:2.0:1788572596.762902:0:70939:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:2.0:1788572596.762905:0:70939:0:(client.c:3112:ptlrpc_free_committed()) lustre-OST0001-osc-ffff94d405cc3800: committing for last_committed 4294968863 gen 19 00000100:00100000:2.0:1788572596.762910:0:70939:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572596.763087:0:70939:0:(obd_config.c:1470:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572596.763094:0:70939:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0001-osc-ffff94d405cc3800 (uuid 9bf287c0-f582-4b83-84c1-15e766218620) 00020000:01000000:2.0:1788572596.763099:0:70939:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0000_UUID : lustre-OST0000-osc-ffff94d405cc3800 00020000:01000000:2.0:1788572596.763101:0:70939:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff94d405cc3800: disconnecting target lustre-OST0000-osc-ffff94d405cc3800 00000020:00000080:1.0:1788572596.763102:0:51031:0:(genops.c:1204:obd_zombie_import_free()) destroying import ffff94d405bab800 for lustre-OST0001-osc-ffff94d405cc3800 00000100:00080000:2.0:1788572596.763109:0:70939:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 9bf287c0-f582-4b83-84c1-15e766218620->lustre-OST0000_UUID 00000020:00000080:1.0:1788572596.763112:0:51031:0:(genops.c:906:class_export_put()) final put ffff94d405ba8000/9bf287c0-f582-4b83-84c1-15e766218620 00000020:00000080:1.0:1788572596.763116:0:51031:0:(genops.c:865:class_export_destroy()) destroying export ffff94d405ba8000/9bf287c0-f582-4b83-84c1-15e766218620 for lustre-OST0001-osc-ffff94d405cc3800 00000020:01000000:1.0:1788572596.763123:0:51031:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0001-osc-ffff94d405cc3800 (9bf287c0-f582-4b83-84c1-15e766218620) 00010000:00010000:2.0:1788572596.763129:0:70939:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x702:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x28400000000 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788572596.763189:0:70939:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 1/0,0 mode: --/PW res: [0x280000401:0x702:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x4829400000020 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788572596.763198:0:70939:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:2.0:1788572596.763234:0:70939:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408529900 going to sleep for 0 seconds 00000100:00100000:2.0:1788572596.763242:0:70939:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job umount:9bf287c0-f582-4b83-84c1-15e766218620:70939:1875450488539520:192.168.202.125@tcp:103:umount.0 00010000:00010000:2.0:1788572596.763253:0:70939:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:2.0:1788572596.763259:0:70939:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:2.0:1788572596.763262:0:70939:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 00000000226484d0/0x1e260726fbbc0b31 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x702:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x4829400000020 nid: local remote: 0xec901abfeb4e09fa expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788572596.763291:0:70939:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x703:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x28400000000 nid: local remote: 0xec901abfeb4e0a08 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572596.763335:0:70939:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 1/0,0 mode: --/PW res: [0x280000401:0x703:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4829400000020 nid: local remote: 0xec901abfeb4e0a08 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572596.763343:0:70939:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:2.0:1788572596.763350:0:70939:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408529900 going to sleep for 0 seconds 00000100:00100000:2.0:1788572596.763356:0:70939:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job umount:9bf287c0-f582-4b83-84c1-15e766218620:70939:1875450488539648:192.168.202.125@tcp:103:umount.0 00010000:00010000:2.0:1788572596.763363:0:70939:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:2.0:1788572596.763369:0:70939:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:2.0:1788572596.763373:0:70939:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff94d405cc3800 lock: 0000000098429620/0x1e260726fbbc0b3f lrc: 0/0,0 mode: --/PW res: [0x280000401:0x703:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4829400000020 nid: local remote: 0xec901abfeb4e0a08 expref: -99 pid: 69530 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00000100:00080000:2.0:1788572596.763481:0:70939:0:(import.c:65:import_set_state_nolock()) ffff94d405bad000 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00100000:3.0:1788572596.763525:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488539776:192.168.202.125@tcp:9:umount.0 00000100:00080000:2.0:1788572596.765166:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bad000 lustre-OST0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:2.0:1788572596.765286:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488539776:192.168.202.125@tcp:9:umount.0 00000100:00080000:0.0:1788572596.765379:0:70939:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:0.0:1788572596.765383:0:70939:0:(client.c:3112:ptlrpc_free_committed()) lustre-OST0000-osc-ffff94d405cc3800: committing for last_committed 4294968865 gen 17 00000100:00100000:0.0:1788572596.765387:0:70939:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572596.765564:0:70939:0:(genops.c:1489:class_disconnect()) disconnect: cookie 0x1e260726fbbbf4f6 00000020:00000080:0.0:1788572596.765582:0:70939:0:(genops.c:906:class_export_put()) final put ffff94d4182fa000/LOV_OSC_UUID 00000020:00000080:1.0:1788572596.765618:0:51031:0:(genops.c:865:class_export_destroy()) destroying export ffff94d4182fa000/LOV_OSC_UUID for lustre-OST0000-osc-ffff94d405cc3800 00000020:01000000:0.0:1788572596.765641:0:70939:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0000-osc-ffff94d405cc3800 (flags='') 00000020:00000080:0.0:1788572596.765647:0:70939:0:(obd_config.c:1470:class_process_config()) processing cmd: cf004 00000020:00000080:0.0:1788572596.765656:0:70939:0:(obd_config.c:955:class_cleanup()) lustre-OST0000-osc-ffff94d405cc3800: forcing exports to disconnect: 0/3 00000020:00080000:0.0:1788572596.765661:0:70939:0:(genops.c:1579:class_disconnect_exports()) OBD device 5 (ffff94d407dbcd40) has no exports 00000008:01000000:0.0:1788572596.765899:0:70939:0:(obd_class.h:671:obd_cleanup_client_import()) lustre-OST0000-osc-ffff94d405cc3800: client import never connected 00000100:00080000:0.0:1788572596.765902:0:70939:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:0.0:1788572596.765905:0:70939:0:(client.c:3112:ptlrpc_free_committed()) lustre-OST0000-osc-ffff94d405cc3800: committing for last_committed 4294968865 gen 18 00000100:00100000:0.0:1788572596.765907:0:70939:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572596.766081:0:51031:0:(genops.c:1204:obd_zombie_import_free()) destroying import ffff94d405bad000 for lustre-OST0000-osc-ffff94d405cc3800 00000020:00000080:0.0:1788572596.766087:0:70939:0:(obd_config.c:1470:class_process_config()) processing cmd: cf002 00000020:00000080:0.0:1788572596.766094:0:70939:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0000-osc-ffff94d405cc3800 (uuid 9bf287c0-f582-4b83-84c1-15e766218620) 00000020:00000080:0.0:1788572596.766096:0:70939:0:(genops.c:906:class_export_put()) final put ffff94d405bad800/9bf287c0-f582-4b83-84c1-15e766218620 00000020:00000080:0.0:1788572596.766098:0:70939:0:(genops.c:865:class_export_destroy()) destroying export ffff94d405bad800/9bf287c0-f582-4b83-84c1-15e766218620 for lustre-OST0000-osc-ffff94d405cc3800 00000020:01000000:0.0:1788572596.766103:0:70939:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0000-osc-ffff94d405cc3800 (9bf287c0-f582-4b83-84c1-15e766218620) 00000020:00000080:0.0:1788572596.766588:0:70939:0:(genops.c:1489:class_disconnect()) disconnect: cookie 0x1e260726fbbbf4ef 00000020:00000080:0.0:1788572596.766594:0:70939:0:(genops.c:906:class_export_put()) final put ffff94d4182ff800/9bf287c0-f582-4b83-84c1-15e766218620 00000020:00000080:1.0:1788572596.766689:0:51031:0:(genops.c:865:class_export_destroy()) destroying export ffff94d4182ff800/9bf287c0-f582-4b83-84c1-15e766218620 for lustre-clilov-ffff94d405cc3800 00000100:00080000:0.0:1788572596.766771:0:70939:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0000_UUID 00010000:00010000:0.0:1788572596.766991:0:70939:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1788572596.767044:0:70939:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1788572596.767052:0:70939:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:0.0:1788572596.767066:0:70939:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405411600 going to sleep for 0 seconds 00000100:00100000:0.0:1788572596.767073:0:70939:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job umount:9bf287c0-f582-4b83-84c1-15e766218620:70939:1875450488539904:192.168.202.125@tcp:103:umount.0 00010000:00010000:0.0:1788572596.767083:0:70939:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:0.0:1788572596.767089:0:70939:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:0.0:1788572596.767092:0:70939:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412536000/0x1e260726fbbc0b9a lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4e0f48 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1788572596.767387:0:70939:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b93 lrc: 2/0,0 mode: PR/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xec901abfeb4e0f10 expref: -99 pid: 70355 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572596.767407:0:70939:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b93 lrc: 1/0,0 mode: --/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xec901abfeb4e0f10 expref: -99 pid: 70355 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572596.767414:0:70939:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:0.0:1788572596.767420:0:70939:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405411600 going to sleep for 0 seconds 00000100:00100000:0.0:1788572596.767425:0:70939:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job umount:9bf287c0-f582-4b83-84c1-15e766218620:70939:1875450488540032:192.168.202.125@tcp:103:umount.0 00010000:00010000:0.0:1788572596.767430:0:70939:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:0.0:1788572596.767434:0:70939:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:0.0:1788572596.767438:0:70939:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412167600/0x1e260726fbbc0b93 lrc: 0/0,0 mode: --/PR res: [0x20000a811:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xec901abfeb4e0f10 expref: -99 pid: 70355 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572596.767505:0:70939:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166a00/0x1e260726fbbc0ba1 lrc: 2/0,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4e0f64 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572596.767523:0:70939:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166a00/0x1e260726fbbc0ba1 lrc: 1/0,0 mode: --/PR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4e0f64 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572596.767529:0:70939:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:0.0:1788572596.767535:0:70939:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405411600 going to sleep for 0 seconds 00000100:00100000:0.0:1788572596.767540:0:70939:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job umount:9bf287c0-f582-4b83-84c1-15e766218620:70939:1875450488540160:192.168.202.125@tcp:103:umount.0 00010000:00010000:0.0:1788572596.767544:0:70939:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:0.0:1788572596.767548:0:70939:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:0.0:1788572596.767550:0:70939:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d405cc3800 lock: ffff94d412166a00/0x1e260726fbbc0ba1 lrc: 0/0,0 mode: --/PR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4e0f64 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00080000:0.0:1788572596.769384:0:70939:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from FULL to CONNECTING 00000100:00100000:2.0:1788572596.769503:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488540288:192.168.202.125@tcp:39:umount.0 00000100:00080000:2.0:1788572596.771545:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405bab000 lustre-MDT0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:2.0:1788572596.771577:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488540288:192.168.202.125@tcp:39:umount.0 00000100:00080000:0.0:1788572596.771640:0:70939:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:0.0:1788572596.771646:0:70939:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 111669149699 gen 5 00000100:00100000:0.0:1788572596.771656:0:70939:0:(client.c:3150:ptlrpc_free_committed()) @@@ free request with old gen req@ffff94d408d3ed80 x1875450488540288/t111669149701(111669149701) o39->lustre-MDT0000-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572606 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572596.771697:0:70939:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572596.773798:0:70939:0:(genops.c:1489:class_disconnect()) disconnect: cookie 0x1e260726fbbbf4e1 00000020:00000080:0.0:1788572596.773814:0:70939:0:(genops.c:906:class_export_put()) final put ffff94d411530000/9bf287c0-f582-4b83-84c1-15e766218620 00000020:00000080:1.0:1788572596.773885:0:51031:0:(genops.c:865:class_export_destroy()) destroying export ffff94d411530000/9bf287c0-f582-4b83-84c1-15e766218620 for lustre-MDT0000-mdc-ffff94d405cc3800 00000100:00080000:0.0:1788572596.773928:0:70939:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 9bf287c0-f582-4b83-84c1-15e766218620->lustre-MDT0001_UUID 00010000:00010000:0.0:1788572596.774335:0:70939:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583000/0x1e260726fbbc0ba8 lrc: 2/0,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x10/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4e0f72 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572596.774387:0:70939:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583000/0x1e260726fbbc0ba8 lrc: 1/0,0 mode: --/PR res: [0x240006991:0xc:0x0].0x0 bits 0x10/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4e0f72 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572596.774393:0:70939:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:0.0:1788572596.774408:0:70939:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d405411600 going to sleep for 0 seconds 00000100:00100000:0.0:1788572596.774418:0:70939:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job umount:9bf287c0-f582-4b83-84c1-15e766218620:70939:1875450488540416:192.168.202.125@tcp:103:umount.0 00010000:00010000:0.0:1788572596.774428:0:70939:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:0.0:1788572596.774437:0:70939:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:0.0:1788572596.774442:0:70939:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff94d405cc3800 lock: ffff94d412583000/0x1e260726fbbc0ba8 lrc: 0/0,0 mode: --/PR res: [0x240006991:0xc:0x0].0x0 bits 0x10/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4e0f72 expref: -99 pid: 70938 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000100:00080000:0.0:1788572596.775338:0:70939:0:(import.c:65:import_set_state_nolock()) ffff94d405baa000 lustre-MDT0001_UUID: changing import state from FULL to CONNECTING 00000100:00100000:2.0:1788572596.775406:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488540544:192.168.202.125@tcp:39:umount.0 00000100:00080000:2.0:1788572596.781900:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405baa000 lustre-MDT0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:2.0:1788572596.781929:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9bf287c0-f582-4b83-84c1-15e766218620:2402:1875450488540544:192.168.202.125@tcp:39:umount.0 00000100:00080000:0.0:1788572596.781956:0:70939:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:0.0:1788572596.781963:0:70939:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff94d405cc3800: committing for last_committed 81604378725 gen 2 00000100:00100000:0.0:1788572596.781976:0:70939:0:(client.c:3150:ptlrpc_free_committed()) @@@ free request with old gen req@ffff94d408d3f480 x1875450488537088/t81604378734(81604378734) o36->lustre-MDT0001-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 568/448 e 0 to 0 dl 1788572612 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572596.782001:0:70939:0:(client.c:3150:ptlrpc_free_committed()) @@@ free request with old gen req@ffff94d4049bdc00 x1875450488538496/t81604378736(81604378736) o36->lustre-MDT0001-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 560/448 e 0 to 0 dl 1788572612 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572596.782015:0:70939:0:(client.c:3150:ptlrpc_free_committed()) @@@ free request with old gen req@ffff94d408d3ed80 x1875450488540544/t81604378738(81604378738) o39->lustre-MDT0001-mdc-ffff94d405cc3800@192.168.202.125@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572606 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572596.782030:0:70939:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572596.785127:0:70939:0:(genops.c:1489:class_disconnect()) disconnect: cookie 0x1e260726fbbbf4e8 00000020:00000080:0.0:1788572596.785142:0:70939:0:(genops.c:906:class_export_put()) final put ffff94d411532000/9bf287c0-f582-4b83-84c1-15e766218620 00000020:00000080:0.0:1788572596.785180:0:70939:0:(genops.c:1489:class_disconnect()) disconnect: cookie 0x1e260726fbbbf4da 00000020:00000080:0.0:1788572596.785184:0:70939:0:(genops.c:906:class_export_put()) final put ffff94d411531000/9bf287c0-f582-4b83-84c1-15e766218620 00000020:00000080:1.0:1788572596.785185:0:51031:0:(genops.c:865:class_export_destroy()) destroying export ffff94d411532000/9bf287c0-f582-4b83-84c1-15e766218620 for lustre-MDT0001-mdc-ffff94d405cc3800 00000020:00000080:1.0:1788572596.785219:0:51031:0:(genops.c:865:class_export_destroy()) destroying export ffff94d411531000/9bf287c0-f582-4b83-84c1-15e766218620 for lustre-clilmv-ffff94d405cc3800 00000020:01000000:0.0:1788572596.785220:0:70939:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilov-ffff94d405cc3800 (flags='') 00000020:00000080:0.0:1788572596.785229:0:70939:0:(obd_config.c:1470:class_process_config()) processing cmd: cf004 00000020:00000080:0.0:1788572596.785238:0:70939:0:(obd_config.c:955:class_cleanup()) lustre-clilov-ffff94d405cc3800: forcing exports to disconnect: 0/1 00000020:00080000:0.0:1788572596.785243:0:70939:0:(genops.c:1579:class_disconnect_exports()) OBD device 1 (ffff94d41290cd40) has no exports 00000020:00000080:0.0:1788572596.785290:0:70939:0:(obd_config.c:1470:class_process_config()) processing cmd: cf002 00000020:00000080:0.0:1788572596.785296:0:70939:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilov-ffff94d405cc3800 (uuid 9bf287c0-f582-4b83-84c1-15e766218620) 00000020:00000080:0.0:1788572596.785299:0:70939:0:(genops.c:906:class_export_put()) final put ffff94d414fac000/9bf287c0-f582-4b83-84c1-15e766218620 00000020:00000080:0.0:1788572596.785301:0:70939:0:(genops.c:865:class_export_destroy()) destroying export ffff94d414fac000/9bf287c0-f582-4b83-84c1-15e766218620 for lustre-clilov-ffff94d405cc3800 00000020:01000000:0.0:1788572596.785304:0:70939:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilov-ffff94d405cc3800 (9bf287c0-f582-4b83-84c1-15e766218620) 00000020:01000000:0.0:1788572596.785546:0:70939:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilmv-ffff94d405cc3800 (flags='') 00000020:00000080:0.0:1788572596.785551:0:70939:0:(obd_config.c:1470:class_process_config()) processing cmd: cf004 00000020:00000080:0.0:1788572596.785554:0:70939:0:(obd_config.c:955:class_cleanup()) lustre-clilmv-ffff94d405cc3800: forcing exports to disconnect: 0/1 00000020:00080000:0.0:1788572596.785556:0:70939:0:(genops.c:1579:class_disconnect_exports()) OBD device 2 (ffff94d41290b380) has no exports 00000020:00000080:0.0:1788572596.785625:0:70939:0:(obd_config.c:1470:class_process_config()) processing cmd: cf002 00000020:00000080:0.0:1788572596.785629:0:70939:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilmv-ffff94d405cc3800 (uuid 9bf287c0-f582-4b83-84c1-15e766218620) 00000020:00000080:0.0:1788572596.785632:0:70939:0:(genops.c:906:class_export_put()) final put ffff94d402e3c800/9bf287c0-f582-4b83-84c1-15e766218620 00000020:00000080:0.0:1788572596.785635:0:70939:0:(genops.c:865:class_export_destroy()) destroying export ffff94d402e3c800/9bf287c0-f582-4b83-84c1-15e766218620 for lustre-clilmv-ffff94d405cc3800 00000020:01000000:0.0:1788572596.785639:0:70939:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilmv-ffff94d405cc3800 (9bf287c0-f582-4b83-84c1-15e766218620) 00000020:01000000:0.0:1788572596.785781:0:70939:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0000-mdc-ffff94d405cc3800 (flags='') 00000020:00000080:0.0:1788572596.785785:0:70939:0:(obd_config.c:1470:class_process_config()) processing cmd: cf004 00000020:00000080:0.0:1788572596.785789:0:70939:0:(obd_config.c:955:class_cleanup()) lustre-MDT0000-mdc-ffff94d405cc3800: forcing exports to disconnect: 0/2 00000020:00080000:0.0:1788572596.785791:0:70939:0:(genops.c:1579:class_disconnect_exports()) OBD device 3 (ffff94d4129099c0) has no exports 00000008:01000000:0.0:1788572596.785959:0:70939:0:(obd_class.h:671:obd_cleanup_client_import()) lustre-MDT0000-mdc-ffff94d405cc3800: client import never connected 00000100:00080000:0.0:1788572596.785964:0:70939:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:0.0:1788572596.785967:0:70939:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d405cc3800: committing for last_committed 111669149699 gen 6 00000100:00100000:0.0:1788572596.785972:0:70939:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572596.788102:0:66171:0:(genops.c:1204:obd_zombie_import_free()) destroying import ffff94d405bab000 for lustre-MDT0000-mdc-ffff94d405cc3800 00000020:00000080:0.0:1788572596.788111:0:70939:0:(obd_config.c:1470:class_process_config()) processing cmd: cf002 00000020:00000080:0.0:1788572596.788119:0:70939:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0000-mdc-ffff94d405cc3800 (uuid 9bf287c0-f582-4b83-84c1-15e766218620) 00000020:00000080:0.0:1788572596.788123:0:70939:0:(genops.c:906:class_export_put()) final put ffff94d402e39800/9bf287c0-f582-4b83-84c1-15e766218620 00000020:00000080:0.0:1788572596.788126:0:70939:0:(genops.c:865:class_export_destroy()) destroying export ffff94d402e39800/9bf287c0-f582-4b83-84c1-15e766218620 for lustre-MDT0000-mdc-ffff94d405cc3800 00000020:01000000:0.0:1788572596.788132:0:70939:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0000-mdc-ffff94d405cc3800 (9bf287c0-f582-4b83-84c1-15e766218620) 00000020:01000000:1.0:1788572596.791554:0:70939:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0001-mdc-ffff94d405cc3800 (flags='') 00000020:00000080:1.0:1788572596.791563:0:70939:0:(obd_config.c:1470:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572596.791570:0:70939:0:(obd_config.c:955:class_cleanup()) lustre-MDT0001-mdc-ffff94d405cc3800: forcing exports to disconnect: 0/2 00000020:00080000:1.0:1788572596.791576:0:70939:0:(genops.c:1579:class_disconnect_exports()) OBD device 4 (ffff94d407db99c0) has no exports 00000008:01000000:1.0:1788572596.791970:0:70939:0:(obd_class.h:671:obd_cleanup_client_import()) lustre-MDT0001-mdc-ffff94d405cc3800: client import never connected 00000100:00080000:1.0:1788572596.791975:0:70939:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:1.0:1788572596.791980:0:70939:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff94d405cc3800: committing for last_committed 81604378725 gen 3 00000100:00100000:1.0:1788572596.791989:0:70939:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572596.796624:0:66171:0:(genops.c:1204:obd_zombie_import_free()) destroying import ffff94d405baa000 for lustre-MDT0001-mdc-ffff94d405cc3800 00000020:00000080:1.0:1788572596.796687:0:70939:0:(obd_config.c:1470:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572596.796701:0:70939:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0001-mdc-ffff94d405cc3800 (uuid 9bf287c0-f582-4b83-84c1-15e766218620) 00000020:00000080:1.0:1788572596.796706:0:70939:0:(genops.c:906:class_export_put()) final put ffff94d405ba8800/9bf287c0-f582-4b83-84c1-15e766218620 00000020:00000080:1.0:1788572596.796710:0:70939:0:(genops.c:865:class_export_destroy()) destroying export ffff94d405ba8800/9bf287c0-f582-4b83-84c1-15e766218620 for lustre-MDT0001-mdc-ffff94d405cc3800 00000020:01000000:1.0:1788572596.796717:0:70939:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0001-mdc-ffff94d405cc3800 (9bf287c0-f582-4b83-84c1-15e766218620) 00000080:02000400:0.0:1788572596.807333:0:70939:0:(llite_lib.c:1748:ll_put_super()) Unmounted lustre-client 00000020:01000000:0.0:1788572596.808852:0:70939:0:(obd_config.c:1197:class_del_profile()) Del profile lustre-client 00000020:01000004:2.0:1788572596.809112:0:70939:0:(obd_mount.c:897:lustre_common_put_super()) dropping sb ffff94d405cc3800 00000100:00080000:2.0:1788572596.809122:0:70939:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import e60431de-58b9-4f69-9048-11a3d82ddcdd->MGS 00010000:00010000:2.0:1788572596.809139:0:70939:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.125@tcp lock: ffff94d412167a00/0x1e260726fbbc0b70 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1008400000000 nid: local remote: 0xec901abfeb4e0ab7 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788572596.809151:0:70939:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.125@tcp lock: ffff94d412167a00/0x1e260726fbbc0b70 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1009400000000 nid: local remote: 0xec901abfeb4e0ab7 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572596.809156:0:70939:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:2.0:1788572596.809159:0:70939:0:(mgc_request.c:954:mgc_blocking_ast()) log lustre-s: disconnecting, won't requeue 10000000:01000000:2.0:1788572596.809161:0:70939:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-sptlrpc 00010000:00010000:2.0:1788572596.809230:0:70939:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.125@tcp lock: ffff94d412167a00/0x1e260726fbbc0b70 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0xec901abfeb4e0ab7 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572596.809237:0:70939:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:2.0:1788572596.809259:0:70939:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408529900 going to sleep for 0 seconds 00000100:00100000:2.0:1788572596.809274:0:70939:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job umount:e60431de-58b9-4f69-9048-11a3d82ddcdd:70939:1875450488540672:192.168.202.125@tcp:103:umount.0 00010000:00010000:2.0:1788572596.809286:0:70939:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:2.0:1788572596.809295:0:70939:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:2.0:1788572596.809299:0:70939:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.125@tcp lock: ffff94d412167a00/0x1e260726fbbc0b70 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0xec901abfeb4e0ab7 expref: -99 pid: 42451 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:2.0:1788572596.809328:0:70939:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from FULL to CONNECTING 00000100:00100000:2.0:1788572596.809574:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e60431de-58b9-4f69-9048-11a3d82ddcdd:2402:1875450488540800:192.168.202.125@tcp:251:umount.0 00000100:00080000:0.0:1788572596.814701:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d40490a800 MGS: changing import state from CONNECTING to CLOSED 10000000:01000000:0.0:1788572596.814714:0:2402:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:0.0:1788572596.814718:0:2402:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:0.0:1788572596.814719:0:2402:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:0.0:1788572596.814770:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e60431de-58b9-4f69-9048-11a3d82ddcdd:2402:1875450488540800:192.168.202.125@tcp:251:umount.0 00000100:00080000:3.0:1788572596.814823:0:70939:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 10000000:01000000:3.0:1788572596.814828:0:70939:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:3.0:1788572596.814832:0:70939:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:3.0:1788572596.814834:0:70939:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00000020:00000080:3.0:1788572596.814846:0:70939:0:(genops.c:1489:class_disconnect()) disconnect: cookie 0x1e260726fbbbf48d 00000020:00000080:3.0:1788572596.814857:0:70939:0:(genops.c:906:class_export_put()) final put ffff94d40490f000/e60431de-58b9-4f69-9048-11a3d82ddcdd 00000020:01000000:3.0:1788572596.814921:0:70939:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of MGC192.168.202.125@tcp (flags='') 00000020:00000080:3.0:1788572596.814929:0:70939:0:(obd_config.c:1470:class_process_config()) processing cmd: cf004 00000020:00000080:3.0:1788572596.814939:0:70939:0:(obd_config.c:955:class_cleanup()) MGC192.168.202.125@tcp: forcing exports to disconnect: 0/3 00000020:00080000:3.0:1788572596.814945:0:70939:0:(genops.c:1579:class_disconnect_exports()) OBD device 0 (ffff94d4113f0000) has no exports 00000020:00000080:1.0:1788572596.815039:0:66171:0:(genops.c:865:class_export_destroy()) destroying export ffff94d40490f000/e60431de-58b9-4f69-9048-11a3d82ddcdd for MGC192.168.202.125@tcp 10000000:01000000:0.0:1788572596.815107:0:42451:0:(mgc_request.c:708:mgc_requeue_thread()) Ending requeue thread 10000000:01000000:3.0:1788572596.815599:0:70939:0:(obd_class.h:671:obd_cleanup_client_import()) MGC192.168.202.125@tcp: client import never connected 00000100:00080000:3.0:1788572596.815602:0:70939:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 10000000:01000000:3.0:1788572596.815604:0:70939:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:3.0:1788572596.815606:0:70939:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:3.0:1788572596.815608:0:70939:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00000020:00000080:3.0:1788572596.815721:0:70939:0:(obd_config.c:1470:class_process_config()) processing cmd: cf002 00000020:00000080:3.0:1788572596.815729:0:70939:0:(obd_config.c:883:class_detach()) detach on obd MGC192.168.202.125@tcp (uuid e60431de-58b9-4f69-9048-11a3d82ddcdd) 00000020:01000004:3.0:1788572596.815735:0:70939:0:(obd_mount.c:710:lustre_put_lsi()) put ffff94d405cc3800 1 00000020:01000004:3.0:1788572596.815738:0:70939:0:(obd_mount.c:668:lustre_free_lsi()) Freeing lsi ffff94d405cc5000 00000020:00000080:1.0:1788572596.815813:0:66171:0:(genops.c:1204:obd_zombie_import_free()) destroying import ffff94d40490a800 for MGC192.168.202.125@tcp 00000020:00000080:1.0:1788572596.815821:0:66171:0:(genops.c:906:class_export_put()) final put ffff94d405cc0800/e60431de-58b9-4f69-9048-11a3d82ddcdd 00000020:00000080:1.0:1788572596.815823:0:66171:0:(genops.c:865:class_export_destroy()) destroying export ffff94d405cc0800/e60431de-58b9-4f69-9048-11a3d82ddcdd for MGC192.168.202.125@tcp 00000020:01000000:1.0:1788572596.815831:0:66171:0:(genops.c:441:class_free_dev()) finishing cleanup of obd MGC192.168.202.125@tcp (e60431de-58b9-4f69-9048-11a3d82ddcdd) 00010000:00010000:2.0:1788572596.816244:0:42449:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-1 blwis in pool) 00010000:00010000:0.0:1788572596.816399:0:63655:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-2 blwis in pool) 00010000:00010000:1.0:1788572596.816528:0:70940:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-3 blwis in pool) 00010000:00010000:0.0:1788572596.816707:0:42448:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-4 blwis in pool) 00000100:00100000:1.0:1788572596.816859:0:42435:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_000: service thread [ffff94d4127e7600:42435] 0 exiting: rc = 1 00000100:00100000:0.0:1788572596.816977:0:42437:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_001: service thread [ffff94d407c36400:42437] 1 exiting: rc = 0 00000100:00100000:2.0:1788572596.817176:0:42439:0:(service.c:3183:ptlrpc_main()) ldlm_cb01_000: service thread [ffff94d405081400:42439] 0 exiting: rc = 1 00000100:00100000:2.0:1788572596.817251:0:42440:0:(service.c:3183:ptlrpc_main()) ldlm_cb01_001: service thread [ffff94d4080d9c00:42440] 1 exiting: rc = 0 00000100:00100000:1.0:1788572596.818347:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572596.818358:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572596.818616:0:42441:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_000: service thread [ffff94d41543dc00:42441] 0 exiting: rc = 1 00000100:00100000:1.0:1788572596.818690:0:42443:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_002: service thread [ffff94d4181b5a00:42443] 2 exiting: rc = 0 00000100:00100000:0.0:1788572596.826018:0:42442:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_001: service thread [ffff94d4181b5800:42442] 1 exiting: rc = 0 00000100:00100000:2.0:1788572596.826128:0:42445:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_001: service thread [ffff94d411bc9a00:42445] 1 exiting: rc = 0 00000100:00100000:2.0:1788572596.826182:0:42447:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_002: service thread [ffff94d4048c5800:42447] 2 exiting: rc = 0 00000100:00100000:3.0:1788572596.826325:0:42444:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_000: service thread [ffff94d412898a00:42444] 0 exiting: rc = 1 00000100:00100000:2.0:1788572596.826377:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826385:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826391:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826397:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826403:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826420:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826426:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826432:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826437:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826443:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826448:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826454:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826459:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826464:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826470:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826476:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826481:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826486:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826491:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826497:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826503:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826509:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826515:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826521:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826526:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826531:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826537:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826543:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826548:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826554:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826561:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826566:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826570:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826575:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826580:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826587:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826592:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826598:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826604:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826610:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826615:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826623:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826628:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826634:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826640:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826646:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826652:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826659:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826664:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826670:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826675:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826694:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826702:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826707:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826715:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826722:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826727:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826732:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826738:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826744:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826750:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826756:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826762:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826769:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826777:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826783:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826788:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826793:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826800:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826806:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826812:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826818:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826823:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826831:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826837:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826843:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826849:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826855:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826861:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826876:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826885:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826891:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826898:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826905:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826911:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826917:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826923:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826929:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826935:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826941:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826952:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826958:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826964:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826970:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826975:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826980:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826986:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826992:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.826998:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.827005:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.827011:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.827018:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.827025:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.827032:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.827039:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.827045:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.827050:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.827055:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.827060:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.827065:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.827070:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.827077:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.827082:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.827088:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.827093:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.827098:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.827103:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.827109:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.827115:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.827120:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.827126:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.827131:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.827136:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.827141:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.827146:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.827151:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.827157:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572596.827162:0:66171:0:(events.c:340:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000020:01000004:0.0:1788572597.350469:0:70961:0:(obd_mount.c:923:lmd_print()) mount data: 00000020:01000004:0.0:1788572597.350474:0:70961:0:(obd_mount.c:925:lmd_print()) profile: lustre-client 00000020:01000004:0.0:1788572597.350476:0:70961:0:(obd_mount.c:926:lmd_print()) device: 192.168.202.125@tcp:/lustre 00000020:01000004:0.0:1788572597.350477:0:70961:0:(obd_mount.c:929:lmd_print()) options: user_xattr,flock 00000080:01200004:0.0:1788572597.350589:0:70961:0:(super25.c:131:lustre_fill_super()) VFS Op: sb ffff94d402e39800 00000080:01000004:0.0:1788572597.350599:0:70961:0:(super25.c:162:lustre_fill_super()) Mounting client lustre-client 00000020:01000004:0.0:1788572597.350616:0:70961:0:(obd_mount.c:411:lustre_start_mgc()) Start MGC 'MGC192.168.202.125@tcp' 00000020:01000004:0.0:1788572597.350646:0:70961:0:(obd_mount.c:230:lustre_start_simple()) Starting OBD MGC192.168.202.125@tcp (typ=mgc) 00000020:00000080:0.0:1788572597.350889:0:70961:0:(genops.c:413:class_newdev()) Allocate new device MGC192.168.202.125@tcp (ffff94d41290b380) 00000020:00000080:0.0:1788572597.350906:0:70961:0:(obd_config.c:720:class_attach_name()) OBD: dev 0 attached type mgc with refcount 2 00000100:00100000:0.0:1788572597.351340:0:70961:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/1 left), rc = 0 00000100:00100000:0.0:1788572597.351367:0:70961:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/1 left), rc = 0 00000100:00100000:0.0:1788572597.351504:0:70961:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[0] started 0 min 2 max 34 00000100:00100000:0.0:1788572597.351510:0:70961:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb00_000' 00000100:00100000:2.0:1788572597.351940:0:70961:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[0] started 1 min 2 max 34 00000100:00100000:2.0:1788572597.351950:0:70961:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb00_001' 00000100:00100000:2.0:1788572597.352298:0:70961:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[1] started 0 min 2 max 34 00000100:00100000:2.0:1788572597.352305:0:70961:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb01_000' 00000100:00100000:0.0:1788572597.352541:0:70961:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[1] started 1 min 2 max 34 00000100:00100000:0.0:1788572597.352548:0:70961:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb01_001' 00000100:00100000:0.0:1788572597.353147:0:70961:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:0.0:1788572597.353790:0:70961:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:0.0:1788572597.353958:0:70961:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 0 min 3 max 34 00000100:00100000:0.0:1788572597.353963:0:70961:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_000' 00000100:00100000:0.0:1788572597.354532:0:70961:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 1 min 3 max 34 00000100:00100000:0.0:1788572597.354538:0:70961:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_001' 00000100:00100000:2.0:1788572597.354888:0:70961:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 2 min 3 max 34 00000100:00100000:2.0:1788572597.354895:0:70961:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_002' 00000100:00100000:2.0:1788572597.355118:0:70961:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 0 min 3 max 34 00000100:00100000:2.0:1788572597.355124:0:70961:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_000' 00000100:00100000:0.0:1788572597.355585:0:70961:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 1 min 3 max 34 00000100:00100000:0.0:1788572597.355591:0:70961:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_001' 00000100:00100000:0.0:1788572597.355826:0:70961:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 2 min 3 max 34 00000100:00100000:0.0:1788572597.355831:0:70961:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_002' 00010000:00010000:2.0:1788572597.356338:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572597.356342:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572597.356344:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572597.356553:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572597.356555:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572597.356557:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00080000:2.0:1788572597.356844:0:70961:0:(ldlm_lib.c:102:import_set_conn()) imp ffff94d405b33800@MGC192.168.202.125@tcp: add connection 192.168.202.125@tcp at head 00010000:00010000:2.0:1788572597.356987:0:70961:0:(ldlm_pool.c:925:ldlm_pool_init()) Lock pool ldlm-pool-MGC192.168.202.125@tcp-0 is initialized 00000040:01000000:2.0:1788572597.356995:0:70961:0:(llog_obd.c:197:llog_setup()) obd MGC192.168.202.125@tcp ctxt 1 is initialized 00010000:00010000:1.0:1788572597.361259:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788572597.361805:0:70976:0:(mgc_request.c:631:mgc_requeue_thread()) Starting requeue thread 00000020:00000080:0.0:1788572597.363367:0:70961:0:(obd_config.c:826:class_setup()) finished setup of obd MGC192.168.202.125@tcp (uuid 2a9ed50a-a289-4e84-96b9-fb493e8051f0) 00000020:00000080:0.0:1788572597.363399:0:70961:0:(genops.c:1406:class_connect()) connect: client 2a9ed50a-a289-4e84-96b9-fb493e8051f0, cookie 0x1e260726fbbc0bd2 00000100:00080000:0.0:1788572597.363417:0:70961:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 0 00000100:00080000:0.0:1788572597.363432:0:70961:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788572597.363472:0:70961:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ed80 x1875450488540928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572597.363532:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488540928:192.168.202.125@tcp:250:mount.lustre.0 00000100:00080000:0.0:1788572597.364029:0:70961:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS 00000080:00200000:0.0:1788572597.364052:0:70961:0:(llite_lib.c:1515:ll_fill_super()) VFS Op: cfg_instance lustre-client-ffff94d402e39800 (sb ffff94d402e39800) 00000080:01000000:0.0:1788572597.364119:0:70961:0:(llite_lib.c:1241:ll_options()) Parsing opts user_xattr,flock 00000080:00000004:0.0:1788572597.364121:0:70961:0:(llite_lib.c:1251:ll_options()) next opt=user_xattr 00000080:00000004:0.0:1788572597.364126:0:70961:0:(llite_lib.c:1251:ll_options()) next opt=flock 00000080:01000000:0.0:1788572597.364134:0:70961:0:(llite_lib.c:1549:ll_fill_super()) llite sb uuid: 86f37580-247f-4c91-a842-1b0c4422007b 00000100:00100000:3.0:1788572597.369150:0:2402:0:(client.c:3112:ptlrpc_free_committed()) MGC192.168.202.125@tcp: committing for last_committed 0 gen 1 00000100:00080000:3.0:1788572597.369170:0:2402:0:(import.c:1132:ptlrpc_connect_interpret()) MGC192.168.202.125@tcp: connect to target with instance 0 00000100:00080000:3.0:1788572597.369180:0:2402:0:(import.c:956:ptlrpc_connect_set_flags()) MGC192.168.202.125@tcp: Resetting ns_connect_flags to server flags: 0xa000011001002060 10000000:01000000:3.0:1788572597.369186:0:2402:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:00080000:3.0:1788572597.369191:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to FULL 10000000:01000000:3.0:1788572597.369195:0:2402:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:3.0:1788572597.369257:0:2402:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:00100000:3.0:1788572597.369270:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488540928:192.168.202.125@tcp:250:mount.lustre.0 10000000:01000000:0.0:1788572597.372727:0:70961:0:(mgc_request.c:1943:mgc_process_config()) parse_log lustre-client from 0 10000000:01000000:0.0:1788572597.372734:0:70961:0:(mgc_request.c:329:config_log_add()) add config log lustre-client-ffff94d402e39800 10000000:01000000:0.0:1788572597.372740:0:70961:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-sptlrpc-ffff94d41290b380 10000000:01000000:0.0:1788572597.372746:0:70961:0:(mgc_request.c:68:mgc_name2resid()) log lustre-sptlrpc to resid 0x65727473756c/0x0 (lustre) 10000000:01000000:0.0:1788572597.372754:0:70961:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-sptlrpc-ffff94d41290b380 from 1 10000000:01000000:0.0:1788572597.372758:0:70961:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:0.0:1788572597.372811:0:70961:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412167000/0x1e260726fbbc0bd9 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 70961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572597.372823:0:70961:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.125@tcp lock: ffff94d412167000/0x1e260726fbbc0bd9 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 70961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572597.372835:0:70961:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.125@tcp lock: ffff94d412167000/0x1e260726fbbc0bd9 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 70961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572597.372862:0:70961:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2a9ed50a-a289-4e84-96b9-fb493e8051f0:70961:1875450488541056:192.168.202.125@tcp:101:mount.lustre.0 00000100:00100000:0.0:1788572597.372964:0:70961:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408b62b40 going to sleep for 11 seconds 00000100:00100000:0.0:1788572597.377049:0:70961:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2a9ed50a-a289-4e84-96b9-fb493e8051f0:70961:1875450488541056:192.168.202.125@tcp:101:mount.lustre.0 00010000:00010000:0.0:1788572597.377085:0:70961:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.125@tcp lock: ffff94d412167000/0x1e260726fbbc0bd9 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0fdb expref: -99 pid: 70961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572597.377099:0:70961:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.125@tcp lock: ffff94d412167000/0x1e260726fbbc0bd9 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xec901abfeb4e0fdb expref: -99 pid: 70961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572597.377179:0:70961:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2a9ed50a-a289-4e84-96b9-fb493e8051f0:70961:1875450488541184:192.168.202.125@tcp:501:mount.lustre.0 00000100:00100000:0.0:1788572597.378325:0:70961:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408b62b40 going to sleep for 11 seconds 00000100:00100000:0.0:1788572597.384757:0:70961:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2a9ed50a-a289-4e84-96b9-fb493e8051f0:70961:1875450488541184:192.168.202.125@tcp:501:mount.lustre.0 10000000:01000000:0.0:1788572597.384815:0:70961:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.202.125@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:0.0:1788572597.384840:0:70961:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412167000/0x1e260726fbbc0bd9 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0fdb expref: -99 pid: 70961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572597.384856:0:70961:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.125@tcp lock: ffff94d412167000/0x1e260726fbbc0bd9 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0fdb expref: -99 pid: 70961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788572597.384866:0:70961:0:(mgc_request.c:194:do_config_log_add()) do adding config log params-ffff94d402e39800 10000000:01000000:0.0:1788572597.384873:0:70961:0:(mgc_request.c:68:mgc_name2resid()) log params to resid 0x736d61726170/0x3 (params) 10000000:01000000:0.0:1788572597.384877:0:70961:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-client-ffff94d402e39800 10000000:01000000:0.0:1788572597.384880:0:70961:0:(mgc_request.c:68:mgc_name2resid()) log lustre-client to resid 0x65727473756c/0x0 (lustre) 10000000:01000000:0.0:1788572597.384884:0:70961:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-cliir-ffff94d402e39800 10000000:01000000:0.0:1788572597.384886:0:70961:0:(mgc_request.c:68:mgc_name2resid()) log lustre-cliir to resid 0x65727473756c/0x2 (lustre) 10000000:01000000:0.0:1788572597.384889:0:70961:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-client-ffff94d402e39800 from 1 10000000:01000000:0.0:1788572597.384892:0:70961:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:0.0:1788572597.384930:0:70961:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412166a00/0x1e260726fbbc0be0 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 70961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572597.384937:0:70961:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.125@tcp lock: ffff94d412166a00/0x1e260726fbbc0be0 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 70961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572597.384948:0:70961:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.125@tcp lock: ffff94d412166a00/0x1e260726fbbc0be0 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 70961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572597.384972:0:70961:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2a9ed50a-a289-4e84-96b9-fb493e8051f0:70961:1875450488541312:192.168.202.125@tcp:101:mount.lustre.0 00000100:00100000:0.0:1788572597.385058:0:70961:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408b62b40 going to sleep for 11 seconds 00000100:00100000:0.0:1788572597.387180:0:70961:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2a9ed50a-a289-4e84-96b9-fb493e8051f0:70961:1875450488541312:192.168.202.125@tcp:101:mount.lustre.0 00010000:00010000:0.0:1788572597.387294:0:70961:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.125@tcp lock: ffff94d412166a00/0x1e260726fbbc0be0 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0fe2 expref: -99 pid: 70961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572597.387308:0:70961:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.125@tcp lock: ffff94d412166a00/0x1e260726fbbc0be0 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xec901abfeb4e0fe2 expref: -99 pid: 70961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572597.387385:0:70961:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2a9ed50a-a289-4e84-96b9-fb493e8051f0:70961:1875450488541440:192.168.202.125@tcp:501:mount.lustre.0 00000100:00100000:0.0:1788572597.387531:0:70961:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408b62b40 going to sleep for 11 seconds 00000100:00100000:0.0:1788572597.390298:0:70961:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2a9ed50a-a289-4e84-96b9-fb493e8051f0:70961:1875450488541440:192.168.202.125@tcp:501:mount.lustre.0 00000100:00100000:0.0:1788572597.390350:0:70961:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2a9ed50a-a289-4e84-96b9-fb493e8051f0:70961:1875450488541568:192.168.202.125@tcp:503:mount.lustre.0 00000100:00100000:0.0:1788572597.390447:0:70961:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408b62b40 going to sleep for 11 seconds 00000100:00100000:0.0:1788572597.391622:0:70961:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2a9ed50a-a289-4e84-96b9-fb493e8051f0:70961:1875450488541568:192.168.202.125@tcp:503:mount.lustre.0 00000100:00100000:2.0:1788572597.392135:0:70980:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job llog_process_th:2a9ed50a-a289-4e84-96b9-fb493e8051f0:70980:1875450488541696:192.168.202.125@tcp:502:llog_process_th.0 00000100:00100000:2.0:1788572597.392356:0:70980:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4156bd780 going to sleep for 11 seconds 00000100:00100000:1.0:1788572597.396374:0:70980:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job llog_process_th:2a9ed50a-a289-4e84-96b9-fb493e8051f0:70980:1875450488541696:192.168.202.125@tcp:502:llog_process_th.0 00000020:01000000:1.0:1788572597.396424:0:70980:0:(obd_config.c:1890:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:1.0:1788572597.396439:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572597.396441:0:70980:0:(obd_config.c:1561:class_process_config()) marker 4 (0x1) lustre-clilov lov setup 00000020:01000000:1.0:1788572597.396449:0:70980:0:(obd_config.c:2005:class_config_llog_handler()) cmd cf001, instance name: lustre-clilov-ffff94d402e39800 00000020:00000080:1.0:1788572597.396452:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf001 00000020:00000080:1.0:1788572597.396467:0:70980:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilov-ffff94d402e39800 (ffff94d404a199c0) 00000020:00000080:1.0:1788572597.396488:0:70980:0:(obd_config.c:720:class_attach_name()) OBD: dev 1 attached type lov with refcount 2 00000020:01000000:1.0:1788572597.396494:0:70980:0:(obd_config.c:2005:class_config_llog_handler()) cmd cf003, instance name: lustre-clilov-ffff94d402e39800 00000020:00000080:1.0:1788572597.396497:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf003 00000020:00000080:1.0:1788572597.396766:0:70980:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-clilov-ffff94d402e39800 (uuid 86f37580-247f-4c91-a842-1b0c4422007b) 00000020:01000000:1.0:1788572597.396771:0:70980:0:(obd_config.c:1890:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:1.0:1788572597.396775:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572597.396776:0:70980:0:(obd_config.c:1561:class_process_config()) marker 4 (0x2) lustre-clilov lov setup 00000020:01000000:1.0:1788572597.396778:0:70980:0:(obd_config.c:1890:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:1.0:1788572597.396780:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572597.396781:0:70980:0:(obd_config.c:1561:class_process_config()) marker 5 (0x1) lustre-clilmv lmv setup 00000020:01000000:1.0:1788572597.396784:0:70980:0:(obd_config.c:2005:class_config_llog_handler()) cmd cf001, instance name: lustre-clilmv-ffff94d402e39800 00000020:00000080:1.0:1788572597.396786:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf001 00000020:00000080:1.0:1788572597.396793:0:70980:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilmv-ffff94d402e39800 (ffff94d404a18000) 00000020:00000080:1.0:1788572597.396803:0:70980:0:(obd_config.c:720:class_attach_name()) OBD: dev 2 attached type lmv with refcount 2 00000020:01000000:1.0:1788572597.396807:0:70980:0:(obd_config.c:2005:class_config_llog_handler()) cmd cf003, instance name: lustre-clilmv-ffff94d402e39800 00000020:00000080:1.0:1788572597.396808:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf003 00000020:00000080:1.0:1788572597.396963:0:70980:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-clilmv-ffff94d402e39800 (uuid 86f37580-247f-4c91-a842-1b0c4422007b) 00000020:01000000:1.0:1788572597.396967:0:70980:0:(obd_config.c:1890:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:1.0:1788572597.396970:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572597.396972:0:70980:0:(obd_config.c:1561:class_process_config()) marker 5 (0x2) lustre-clilmv lmv setup 00000020:01000000:1.0:1788572597.396974:0:70980:0:(obd_config.c:1890:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:1.0:1788572597.396976:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572597.396977:0:70980:0:(obd_config.c:1561:class_process_config()) marker 6 (0x1) lustre-client mount opts 00000020:00000080:1.0:1788572597.396980:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf007 00000020:00000080:1.0:1788572597.396982:0:70980:0:(obd_config.c:1514:class_process_config()) mountopt: profile lustre-client osc lustre-clilov mdc lustre-clilmv 00000020:01000000:1.0:1788572597.396985:0:70980:0:(obd_config.c:1148:class_add_profile()) Add profile lustre-client 00000020:01000000:1.0:1788572597.396990:0:70980:0:(obd_config.c:1890:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:1.0:1788572597.396992:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572597.396993:0:70980:0:(obd_config.c:1561:class_process_config()) marker 6 (0x2) lustre-client mount opts 00000020:01000000:1.0:1788572597.396995:0:70980:0:(obd_config.c:1890:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:1.0:1788572597.396998:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572597.396999:0:70980:0:(obd_config.c:1561:class_process_config()) marker 7 (0x1) lustre-MDT0000 add mdc 00000020:00000080:1.0:1788572597.397002:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf005 00000020:00000080:1.0:1788572597.397006:0:70980:0:(obd_config.c:1485:class_process_config()) adding mapping from uuid 192.168.202.125@tcp to nid 0x20000c0a8ca7d (192.168.202.125@tcp) 00000020:01000000:1.0:1788572597.397028:0:70980:0:(obd_config.c:2005:class_config_llog_handler()) cmd cf001, instance name: lustre-MDT0000-mdc-ffff94d402e39800 00000020:00000080:1.0:1788572597.397030:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf001 00000020:00000080:1.0:1788572597.397037:0:70980:0:(genops.c:413:class_newdev()) Allocate new device lustre-MDT0000-mdc-ffff94d402e39800 (ffff94d404a1b380) 00000020:00000080:1.0:1788572597.397043:0:70980:0:(obd_config.c:720:class_attach_name()) OBD: dev 3 attached type mdc with refcount 2 00000020:01000000:1.0:1788572597.397047:0:70980:0:(obd_config.c:2005:class_config_llog_handler()) cmd cf003, instance name: lustre-MDT0000-mdc-ffff94d402e39800 00000020:00000080:1.0:1788572597.397050:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf003 00010000:00080000:1.0:1788572597.397105:0:70980:0:(ldlm_lib.c:102:import_set_conn()) imp ffff94d411203000@lustre-MDT0000-mdc-ffff94d402e39800: add connection 192.168.202.125@tcp at head 00010000:00010000:1.0:1788572597.399037:0:70980:0:(ldlm_pool.c:925:ldlm_pool_init()) Lock pool ldlm-pool-lustre-MDT0000-mdc-ffff94d402e39800-1 is initialized 00000040:01000000:1.0:1788572597.399494:0:70980:0:(llog_obd.c:197:llog_setup()) obd lustre-MDT0000-mdc-ffff94d402e39800 ctxt 13 is initialized 00000020:00000080:3.0:1788572597.399789:0:70980:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-MDT0000-mdc-ffff94d402e39800 (uuid 86f37580-247f-4c91-a842-1b0c4422007b) 00000020:01000000:3.0:1788572597.399807:0:70980:0:(obd_config.c:2005:class_config_llog_handler()) cmd cf014, instance name: lustre-clilmv-ffff94d402e39800 00000020:00000080:3.0:1788572597.399812:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf014 00800000:01000000:3.0:1788572597.399836:0:70980:0:(lmv_obd.c:442:lmv_add_target()) Target uuid: lustre-MDT0000_UUID. index 0 00000020:01000000:3.0:1788572597.399872:0:70980:0:(obd_config.c:1890:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788572597.399876:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572597.399878:0:70980:0:(obd_config.c:1561:class_process_config()) marker 7 (0x2) lustre-MDT0000 add mdc 00000020:01000000:3.0:1788572597.399884:0:70980:0:(obd_config.c:1890:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:3.0:1788572597.399888:0:70980:0:(obd_config.c:1899:class_config_llog_handler()) SKIP #9 00000020:01000000:3.0:1788572597.399890:0:70980:0:(obd_config.c:1925:class_config_llog_handler()) skipping 0x6 00000020:01000000:3.0:1788572597.399892:0:70980:0:(obd_config.c:1925:class_config_llog_handler()) skipping 0x6 00000020:01000000:3.0:1788572597.399895:0:70980:0:(obd_config.c:1890:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:3.0:1788572597.399898:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572597.399901:0:70980:0:(obd_config.c:1561:class_process_config()) marker 9 (0x6) lustre sys.timeout 00000020:01000000:3.0:1788572597.399905:0:70980:0:(obd_config.c:1890:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788572597.399910:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572597.399911:0:70980:0:(obd_config.c:1561:class_process_config()) marker 16 (0x1) lustre-MDT0001 add mdc 00000020:00000080:3.0:1788572597.399915:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf005 00000020:00000080:3.0:1788572597.399921:0:70980:0:(obd_config.c:1485:class_process_config()) adding mapping from uuid 192.168.202.125@tcp to nid 0x20000c0a8ca7d (192.168.202.125@tcp) 00000020:01000000:3.0:1788572597.399946:0:70980:0:(obd_config.c:2005:class_config_llog_handler()) cmd cf001, instance name: lustre-MDT0001-mdc-ffff94d402e39800 00000020:00000080:3.0:1788572597.399949:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf001 00000020:00000080:3.0:1788572597.399959:0:70980:0:(genops.c:413:class_newdev()) Allocate new device lustre-MDT0001-mdc-ffff94d402e39800 (ffff94d4113f4d40) 00000020:00000080:3.0:1788572597.399977:0:70980:0:(obd_config.c:720:class_attach_name()) OBD: dev 4 attached type mdc with refcount 2 00000020:01000000:3.0:1788572597.399983:0:70980:0:(obd_config.c:2005:class_config_llog_handler()) cmd cf003, instance name: lustre-MDT0001-mdc-ffff94d402e39800 00000020:00000080:3.0:1788572597.399986:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf003 00010000:00080000:3.0:1788572597.400050:0:70980:0:(ldlm_lib.c:102:import_set_conn()) imp ffff94d405c94000@lustre-MDT0001-mdc-ffff94d402e39800: add connection 192.168.202.125@tcp at head 00010000:00010000:3.0:1788572597.402131:0:70980:0:(ldlm_pool.c:925:ldlm_pool_init()) Lock pool ldlm-pool-lustre-MDT0001-mdc-ffff94d402e39800-2 is initialized 00000040:01000000:3.0:1788572597.402447:0:70980:0:(llog_obd.c:197:llog_setup()) obd lustre-MDT0001-mdc-ffff94d402e39800 ctxt 13 is initialized 00000020:00000080:3.0:1788572597.402597:0:70980:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-MDT0001-mdc-ffff94d402e39800 (uuid 86f37580-247f-4c91-a842-1b0c4422007b) 00000020:01000000:3.0:1788572597.402609:0:70980:0:(obd_config.c:2005:class_config_llog_handler()) cmd cf014, instance name: lustre-clilmv-ffff94d402e39800 00000020:00000080:3.0:1788572597.402613:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf014 00800000:01000000:3.0:1788572597.402632:0:70980:0:(lmv_obd.c:442:lmv_add_target()) Target uuid: lustre-MDT0001_UUID. index 1 00000020:01000000:3.0:1788572597.402755:0:70980:0:(obd_config.c:1890:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788572597.402760:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572597.402762:0:70980:0:(obd_config.c:1561:class_process_config()) marker 16 (0x2) lustre-MDT0001 add mdc 00000020:01000000:3.0:1788572597.402766:0:70980:0:(obd_config.c:1890:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:3.0:1788572597.402769:0:70980:0:(obd_config.c:1899:class_config_llog_handler()) SKIP #20 00000020:01000000:3.0:1788572597.402771:0:70980:0:(obd_config.c:1925:class_config_llog_handler()) skipping 0x6 00000020:01000000:3.0:1788572597.402773:0:70980:0:(obd_config.c:1925:class_config_llog_handler()) skipping 0x6 00000020:01000000:3.0:1788572597.402775:0:70980:0:(obd_config.c:1890:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:3.0:1788572597.402778:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572597.402780:0:70980:0:(obd_config.c:1561:class_process_config()) marker 20 (0x6) lustre sys.timeout 00000020:01000000:3.0:1788572597.402783:0:70980:0:(obd_config.c:1890:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:3.0:1788572597.402788:0:70980:0:(obd_mount.c:961:lustre_check_exclusion()) Check exclusion lustre-OST0000 (0) in 0 of 192.168.202.125@tcp:/lustre 00000020:00000080:3.0:1788572597.402791:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572597.402793:0:70980:0:(obd_config.c:1561:class_process_config()) marker 24 (0x1) lustre-OST0000 add osc 00000020:00000080:3.0:1788572597.402796:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf005 00000020:00000080:3.0:1788572597.402802:0:70980:0:(obd_config.c:1485:class_process_config()) adding mapping from uuid 192.168.202.125@tcp to nid 0x20000c0a8ca7d (192.168.202.125@tcp) 00000020:01000000:3.0:1788572597.402825:0:70980:0:(obd_config.c:2005:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0000-osc-ffff94d402e39800 00000020:00000080:3.0:1788572597.402828:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf001 00000020:00000080:3.0:1788572597.402839:0:70980:0:(genops.c:413:class_newdev()) Allocate new device lustre-OST0000-osc-ffff94d402e39800 (ffff94d4113f19c0) 00000020:00000080:3.0:1788572597.402852:0:70980:0:(obd_config.c:720:class_attach_name()) OBD: dev 5 attached type osc with refcount 2 00000020:01000000:3.0:1788572597.402858:0:70980:0:(obd_config.c:2005:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0000-osc-ffff94d402e39800 00000020:00000080:3.0:1788572597.402862:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf003 00010000:00080000:3.0:1788572597.402920:0:70980:0:(ldlm_lib.c:102:import_set_conn()) imp ffff94d405c96800@lustre-OST0000-osc-ffff94d402e39800: add connection 192.168.202.125@tcp at head 00010000:00010000:3.0:1788572597.403137:0:70980:0:(ldlm_pool.c:925:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0000-osc-ffff94d402e39800-3 is initialized 00000020:00000080:3.0:1788572597.403439:0:70980:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-OST0000-osc-ffff94d402e39800 (uuid 86f37580-247f-4c91-a842-1b0c4422007b) 00000020:01000000:3.0:1788572597.403447:0:70980:0:(obd_config.c:2005:class_config_llog_handler()) cmd cf00d, instance name: lustre-clilov-ffff94d402e39800 00000020:00000080:3.0:1788572597.403451:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf00d 00020000:01000000:3.0:1788572597.403464:0:70980:0:(lov_obd.c:481:lov_add_target()) uuid:lustre-OST0000_UUID idx:0 gen:1 active:1 00020000:01000000:3.0:1788572597.403471:0:70980:0:(lov_obd.c:510:lov_add_target()) lustre-clilov-ffff94d402e39800: Adding target lustre-OST0000_UUID 00020000:01000000:3.0:1788572597.403477:0:70980:0:(lov_obd.c:532:lov_add_target()) idx=0 ltd_gen=1 ld_tgt_count=1 00000020:01000000:3.0:1788572597.403491:0:70980:0:(obd_config.c:1890:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788572597.403495:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572597.403498:0:70980:0:(obd_config.c:1561:class_process_config()) marker 24 (0x2) lustre-OST0000 add osc 00000020:01000000:3.0:1788572597.403501:0:70980:0:(obd_config.c:1890:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:3.0:1788572597.403504:0:70980:0:(obd_config.c:1899:class_config_llog_handler()) SKIP #27 00000020:01000000:3.0:1788572597.403505:0:70980:0:(obd_config.c:1925:class_config_llog_handler()) skipping 0x6 00000020:01000000:3.0:1788572597.403507:0:70980:0:(obd_config.c:1925:class_config_llog_handler()) skipping 0x6 00000020:01000000:3.0:1788572597.403509:0:70980:0:(obd_config.c:1890:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:3.0:1788572597.403513:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572597.403581:0:70980:0:(obd_config.c:1561:class_process_config()) marker 27 (0x6) lustre sys.timeout 00000020:01000000:3.0:1788572597.403589:0:70980:0:(obd_config.c:1890:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:3.0:1788572597.403592:0:70980:0:(obd_mount.c:961:lustre_check_exclusion()) Check exclusion lustre-OST0001 (1) in 0 of 192.168.202.125@tcp:/lustre 00000020:00000080:3.0:1788572597.403597:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572597.403599:0:70980:0:(obd_config.c:1561:class_process_config()) marker 32 (0x1) lustre-OST0001 add osc 00000020:00000080:3.0:1788572597.403604:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf005 00000020:00000080:3.0:1788572597.403607:0:70980:0:(obd_config.c:1485:class_process_config()) adding mapping from uuid 192.168.202.125@tcp to nid 0x20000c0a8ca7d (192.168.202.125@tcp) 00000020:01000000:3.0:1788572597.403623:0:70980:0:(obd_config.c:2005:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0001-osc-ffff94d402e39800 00000020:00000080:3.0:1788572597.403626:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf001 00000020:00000080:3.0:1788572597.403634:0:70980:0:(genops.c:413:class_newdev()) Allocate new device lustre-OST0001-osc-ffff94d402e39800 (ffff94d4113f3380) 00000020:00000080:3.0:1788572597.403646:0:70980:0:(obd_config.c:720:class_attach_name()) OBD: dev 6 attached type osc with refcount 2 00000020:01000000:3.0:1788572597.403653:0:70980:0:(obd_config.c:2005:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0001-osc-ffff94d402e39800 00000020:00000080:3.0:1788572597.403666:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf003 00010000:00080000:3.0:1788572597.403716:0:70980:0:(ldlm_lib.c:102:import_set_conn()) imp ffff94d405c93800@lustre-OST0001-osc-ffff94d402e39800: add connection 192.168.202.125@tcp at head 00010000:00010000:3.0:1788572597.404024:0:70980:0:(ldlm_pool.c:925:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0001-osc-ffff94d402e39800-4 is initialized 00000020:00000080:3.0:1788572597.404348:0:70980:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-OST0001-osc-ffff94d402e39800 (uuid 86f37580-247f-4c91-a842-1b0c4422007b) 00000020:01000000:3.0:1788572597.404357:0:70980:0:(obd_config.c:2005:class_config_llog_handler()) cmd cf00d, instance name: lustre-clilov-ffff94d402e39800 00000020:00000080:3.0:1788572597.404362:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf00d 00020000:01000000:3.0:1788572597.404372:0:70980:0:(lov_obd.c:481:lov_add_target()) uuid:lustre-OST0001_UUID idx:1 gen:1 active:1 00020000:01000000:3.0:1788572597.404378:0:70980:0:(lov_obd.c:510:lov_add_target()) lustre-clilov-ffff94d402e39800: Adding target lustre-OST0001_UUID 00020000:01000000:3.0:1788572597.404382:0:70980:0:(lov_obd.c:532:lov_add_target()) idx=1 ltd_gen=1 ld_tgt_count=2 00000020:01000000:3.0:1788572597.404389:0:70980:0:(obd_config.c:1890:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788572597.404393:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572597.404395:0:70980:0:(obd_config.c:1561:class_process_config()) marker 32 (0x2) lustre-OST0001 add osc 00000020:01000000:3.0:1788572597.404399:0:70980:0:(obd_config.c:1890:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788572597.404403:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572597.404405:0:70980:0:(obd_config.c:1561:class_process_config()) marker 36 (0x1) lustre sys.timeout 00000020:00000080:3.0:1788572597.404409:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf009 00000020:00000080:3.0:1788572597.404412:0:70980:0:(obd_config.c:1535:class_process_config()) changing lustre timeout from 20 to 20 00000020:01000000:3.0:1788572597.404415:0:70980:0:(obd_config.c:1890:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788572597.404418:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572597.404419:0:70980:0:(obd_config.c:1561:class_process_config()) marker 36 (0x2) lustre sys.timeout 00000100:00100000:3.0:1788572597.404461:0:70980:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job llog_process_th:2a9ed50a-a289-4e84-96b9-fb493e8051f0:70980:1875450488541824:192.168.202.125@tcp:502:llog_process_th.0 00000100:00100000:3.0:1788572597.404725:0:70980:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411806f00 going to sleep for 11 seconds 00000100:00100000:3.0:1788572597.407256:0:70980:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job llog_process_th:2a9ed50a-a289-4e84-96b9-fb493e8051f0:70980:1875450488541824:192.168.202.125@tcp:502:llog_process_th.0 00000020:01000000:3.0:1788572597.407297:0:70980:0:(obd_config.c:1890:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788572597.407308:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572597.407310:0:70980:0:(obd_config.c:1561:class_process_config()) marker 41 (0x1) lustre sys.jobid_var 00000020:00000080:3.0:1788572597.407317:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf00f 00000100:00100000:3.0:1788572597.407353:0:70980:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job llog_process_th:2a9ed50a-a289-4e84-96b9-fb493e8051f0:70980:1875450488541952:192.168.202.125@tcp:502:llog_process_th.0 00000100:00100000:3.0:1788572597.407512:0:70980:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411806f00 going to sleep for 11 seconds 00000100:00100000:3.0:1788572597.410982:0:70980:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job llog_process_th:2a9ed50a-a289-4e84-96b9-fb493e8051f0:70980:1875450488541952:192.168.202.125@tcp:502:llog_process_th.0 00000020:01000000:3.0:1788572597.411024:0:70980:0:(obd_config.c:1890:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788572597.411032:0:70980:0:(obd_config.c:1470:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572597.411034:0:70980:0:(obd_config.c:1561:class_process_config()) marker 41 (0x2) lustre sys.jobid_var 00000040:00080000:3.0:1788572597.411039:0:70980:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 52 count 51 00000020:01000000:0.0:1788572597.411119:0:70961:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 1-51 (rc=0) 10000000:01000000:0.0:1788572597.411143:0:70961:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.202.125@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:0.0:1788572597.411164:0:70961:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412166a00/0x1e260726fbbc0be0 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0fe2 expref: -99 pid: 70961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572597.411176:0:70961:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.125@tcp lock: ffff94d412166a00/0x1e260726fbbc0be0 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0fe2 expref: -99 pid: 70961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788572597.411185:0:70961:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-cliir-ffff94d402e39800 from 1 10000000:01000000:0.0:1788572597.411187:0:70961:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788572597.411235:0:70961:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412166e00/0x1e260726fbbc0c11 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 70961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572597.411243:0:70961:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.125@tcp lock: ffff94d412166e00/0x1e260726fbbc0c11 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 70961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572597.411253:0:70961:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.125@tcp lock: ffff94d412166e00/0x1e260726fbbc0c11 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 70961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572597.411278:0:70961:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2a9ed50a-a289-4e84-96b9-fb493e8051f0:70961:1875450488542080:192.168.202.125@tcp:101:mount.lustre.0 00000100:00100000:0.0:1788572597.411352:0:70961:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572597.418815:0:70961:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2a9ed50a-a289-4e84-96b9-fb493e8051f0:70961:1875450488542080:192.168.202.125@tcp:101:mount.lustre.0 00010000:00010000:2.0:1788572597.418843:0:70961:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.125@tcp lock: ffff94d412166e00/0x1e260726fbbc0c11 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0fe9 expref: -99 pid: 70961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572597.418856:0:70961:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.125@tcp lock: ffff94d412166e00/0x1e260726fbbc0c11 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xec901abfeb4e0fe9 expref: -99 pid: 70961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572597.419566:0:70961:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2a9ed50a-a289-4e84-96b9-fb493e8051f0:70961:1875450488542208:192.168.202.125@tcp:256:mount.lustre.0 00000100:00100000:2.0:1788572597.419645:0:70961:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4156bdf00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572597.422956:0:70961:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2a9ed50a-a289-4e84-96b9-fb493e8051f0:70961:1875450488542208:192.168.202.125@tcp:256:mount.lustre.0 10000000:01000000:0.0:1788572597.423614:0:70961:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.202.125@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:0.0:1788572597.423637:0:70961:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412166e00/0x1e260726fbbc0c11 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0fe9 expref: -99 pid: 70961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572597.423667:0:70961:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.125@tcp lock: ffff94d412166e00/0x1e260726fbbc0c11 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0fe9 expref: -99 pid: 70961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788572597.423676:0:70961:0:(mgc_request.c:1801:mgc_process_log()) Process log params-ffff94d402e39800 from 1 10000000:01000000:0.0:1788572597.423679:0:70961:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:0.0:1788572597.423714:0:70961:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412167a00/0x1e260726fbbc0c18 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 70961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572597.423721:0:70961:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.125@tcp lock: ffff94d412167a00/0x1e260726fbbc0c18 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 70961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572597.423735:0:70961:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.125@tcp lock: ffff94d412167a00/0x1e260726fbbc0c18 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 70961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572597.423756:0:70961:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3fb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2a9ed50a-a289-4e84-96b9-fb493e8051f0:70961:1875450488542336:192.168.202.125@tcp:101:mount.lustre.0 00000100:00100000:0.0:1788572597.423820:0:70961:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 11 seconds 00000100:00100000:0.0:1788572597.424922:0:70961:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3fb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2a9ed50a-a289-4e84-96b9-fb493e8051f0:70961:1875450488542336:192.168.202.125@tcp:101:mount.lustre.0 00010000:00010000:0.0:1788572597.424942:0:70961:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.125@tcp lock: ffff94d412167a00/0x1e260726fbbc0c18 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0ff0 expref: -99 pid: 70961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572597.424953:0:70961:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.125@tcp lock: ffff94d412167a00/0x1e260726fbbc0c18 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xec901abfeb4e0ff0 expref: -99 pid: 70961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572597.425012:0:70961:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3fb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2a9ed50a-a289-4e84-96b9-fb493e8051f0:70961:1875450488542464:192.168.202.125@tcp:501:mount.lustre.0 00000100:00100000:0.0:1788572597.425061:0:70961:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 11 seconds 00000100:00100000:0.0:1788572597.426032:0:70961:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3fb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2a9ed50a-a289-4e84-96b9-fb493e8051f0:70961:1875450488542464:192.168.202.125@tcp:501:mount.lustre.0 00000100:00100000:0.0:1788572597.426064:0:70961:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3fb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2a9ed50a-a289-4e84-96b9-fb493e8051f0:70961:1875450488542592:192.168.202.125@tcp:503:mount.lustre.0 00000100:00100000:0.0:1788572597.426119:0:70961:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 11 seconds 00000100:00100000:0.0:1788572597.427002:0:70961:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3fb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2a9ed50a-a289-4e84-96b9-fb493e8051f0:70961:1875450488542592:192.168.202.125@tcp:503:mount.lustre.0 00000100:00100000:3.0:1788572597.427404:0:70985:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job llog_process_th:2a9ed50a-a289-4e84-96b9-fb493e8051f0:70985:1875450488542720:192.168.202.125@tcp:502:llog_process_th.0 00000100:00100000:3.0:1788572597.427472:0:70985:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d411806840 going to sleep for 11 seconds 00000100:00100000:1.0:1788572597.428969:0:70985:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job llog_process_th:2a9ed50a-a289-4e84-96b9-fb493e8051f0:70985:1875450488542720:192.168.202.125@tcp:502:llog_process_th.0 00000020:01000000:1.0:1788572597.428997:0:70985:0:(obd_config.c:1890:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:1.0:1788572597.429005:0:70985:0:(obd_config.c:1470:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572597.429006:0:70985:0:(obd_config.c:1561:class_process_config()) marker 2 (0x1) general debug_raw_pointers 00000020:01000000:1.0:1788572597.429011:0:70985:0:(obd_config.c:2005:class_config_llog_handler()) cmd ce032, instance name: general-ffff94d402e39800 00000020:00000080:1.0:1788572597.429013:0:70985:0:(obd_config.c:1470:class_process_config()) processing cmd: ce032 00000020:01000000:1.0:1788572597.429021:0:70985:0:(obd_config.c:1890:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:1.0:1788572597.429023:0:70985:0:(obd_config.c:1470:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572597.429025:0:70985:0:(obd_config.c:1561:class_process_config()) marker 2 (0x2) general debug_raw_pointers 00000020:01000000:1.0:1788572597.429027:0:70985:0:(obd_config.c:1890:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:1.0:1788572597.429029:0:70985:0:(obd_config.c:1470:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572597.429030:0:70985:0:(obd_config.c:1561:class_process_config()) marker 3 (0x1) general lod.*.mdt_hash 00000020:01000000:1.0:1788572597.429033:0:70985:0:(obd_config.c:2005:class_config_llog_handler()) cmd ce032, instance name: general-ffff94d402e39800 00000020:00000080:1.0:1788572597.429035:0:70985:0:(obd_config.c:1470:class_process_config()) processing cmd: ce032 00000100:00100000:1.0:1788572597.429124:0:70985:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job llog_process_th:2a9ed50a-a289-4e84-96b9-fb493e8051f0:70985:1875450488542848:192.168.202.125@tcp:502:llog_process_th.0 00000100:00100000:1.0:1788572597.429177:0:70985:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408caff00 going to sleep for 11 seconds 00000100:00100000:1.0:1788572597.430058:0:70985:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job llog_process_th:2a9ed50a-a289-4e84-96b9-fb493e8051f0:70985:1875450488542848:192.168.202.125@tcp:502:llog_process_th.0 00000020:01000000:1.0:1788572597.430083:0:70985:0:(obd_config.c:1890:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:1.0:1788572597.430089:0:70985:0:(obd_config.c:1470:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572597.430090:0:70985:0:(obd_config.c:1561:class_process_config()) marker 3 (0x2) general lod.*.mdt_hash 00000040:00080000:1.0:1788572597.430094:0:70985:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 7 count 7 00000020:01000000:2.0:1788572597.430190:0:70961:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 1-6 (rc=0) 10000000:01000000:2.0:1788572597.430237:0:70961:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.202.125@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:2.0:1788572597.430261:0:70961:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.125@tcp lock: ffff94d412167a00/0x1e260726fbbc0c18 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0ff0 expref: -99 pid: 70961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572597.430278:0:70961:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.125@tcp lock: ffff94d412167a00/0x1e260726fbbc0c18 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xec901abfeb4e0ff0 expref: -99 pid: 70961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:01000000:2.0:1788572597.430304:0:70961:0:(llite_lib.c:1623:ll_fill_super()) Found profile lustre-client: mdc=lustre-clilmv osc=lustre-clilov 00000020:00000080:2.0:1788572597.430328:0:70961:0:(genops.c:1406:class_connect()) connect: client 86f37580-247f-4c91-a842-1b0c4422007b, cookie 0x1e260726fbbc0c1f 00800000:01000000:2.0:1788572597.430350:0:70961:0:(lmv_obd.c:515:lmv_check_connect()) Time to connect 86f37580-247f-4c91-a842-1b0c4422007b to lustre-clilmv-ffff94d402e39800 00800000:01000000:2.0:1788572597.430356:0:70961:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0000-mdc-ffff94d402e39800(86f37580-247f-4c91-a842-1b0c4422007b) - lustre-MDT0000_UUID, 86f37580-247f-4c91-a842-1b0c4422007b 00000020:00000080:2.0:1788572597.430366:0:70961:0:(genops.c:1406:class_connect()) connect: client 86f37580-247f-4c91-a842-1b0c4422007b, cookie 0x1e260726fbbc0c26 00000100:00080000:2.0:1788572597.430373:0:70961:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 0 00000100:00080000:2.0:1788572597.430379:0:70961:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572597.430390:0:70961:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 134217728, lost: 0. 00000100:00100000:2.0:1788572597.430410:0:70961:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60700 x1875450488542976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572597.430437:0:70961:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID 00800000:01000000:2.0:1788572597.430510:0:70961:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0000-mdc-ffff94d402e39800(86f37580-247f-4c91-a842-1b0c4422007b) successfully (5) 00800000:00000004:2.0:1788572597.430514:0:70961:0:(obd_class.h:1057:obd_statfs_async()) lustre-MDT0000-mdc-ffff94d402e39800: age 2750, max_age 0 00800000:01000000:2.0:1788572597.430545:0:70961:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0001-mdc-ffff94d402e39800(86f37580-247f-4c91-a842-1b0c4422007b) - lustre-MDT0001_UUID, 86f37580-247f-4c91-a842-1b0c4422007b 00000020:00000080:2.0:1788572597.430552:0:70961:0:(genops.c:1406:class_connect()) connect: client 86f37580-247f-4c91-a842-1b0c4422007b, cookie 0x1e260726fbbc0c2d 00000100:00080000:2.0:1788572597.430557:0:70961:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 0 00000100:00080000:2.0:1788572597.430562:0:70961:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572597.430568:0:70961:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 134217728, lost: 0. 00000100:00100000:2.0:1788572597.430579:0:70961:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488543232/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572597.430588:0:70961:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID 00800000:01000000:2.0:1788572597.430669:0:70961:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0001-mdc-ffff94d402e39800(86f37580-247f-4c91-a842-1b0c4422007b) successfully (5) 00800000:00000004:2.0:1788572597.430673:0:70961:0:(obd_class.h:1057:obd_statfs_async()) lustre-MDT0001-mdc-ffff94d402e39800: age 2750, max_age 0 00000080:00000004:2.0:1788572597.430691:0:70961:0:(obd_class.h:1091:obd_statfs()) lustre-clilmv-ffff94d402e39800: age 2750, max_age 3749 00800000:00000004:2.0:1788572597.430696:0:70961:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0000-mdc-ffff94d402e39800: age 2750, max_age 3749 00000100:00080000:2.0:1788572597.430710:0:70961:0:(client.c:1832:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff94d408c60000 x1875450488543488/t0(0) o41->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572597.430719:0:70961:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4156bd180 going to sleep for 0 seconds 00000100:00100000:3.0:1788572597.431760:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488542976:192.168.202.125@tcp:38:mount.lustre.0 00000100:00100000:3.0:1788572597.431854:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488543232:192.168.202.125@tcp:38:mount.lustre.0 00000100:00080000:3.0:1788572597.431927:0:2405:0:(client.c:1832:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff94d408c62d80 x1875450488543104/t0(0) o41->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572597.431944:0:2405:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00080000:3.0:1788572597.431950:0:2405:0:(client.c:1832:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff94d408c61180 x1875450488543360/t0(0) o41->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572597.433767:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff94d402e39800: committing for last_committed 0 gen 1 00000100:00080000:0.0:1788572597.433779:0:2402:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff94d402e39800: connect to target with instance 26 00000100:00080000:0.0:1788572597.433787:0:2402:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-MDT0000-mdc-ffff94d402e39800: Resetting ns_connect_flags to server flags: 0xae7a7e7be344d3b8 00000100:00080000:0.0:1788572597.433796:0:2402:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-MDT0000_UUID 00000100:00080000:0.0:1788572597.433799:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to FULL 00000100:00080000:3.0:1788572597.434000:0:2402:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000ba1d2706): req@ffff94d408c60000 x1875450488543488/t0(0) o41->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572597.434023:0:2402:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000af1897cb): req@ffff94d408c62d80 x1875450488543104/t0(0) o41->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572597.434039:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488542976:192.168.202.125@tcp:38:mount.lustre.0 00000100:00100000:3.0:1788572597.434058:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff94d402e39800: committing for last_committed 0 gen 1 00000100:00080000:3.0:1788572597.434064:0:2402:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0001-mdc-ffff94d402e39800: connect to target with instance 19 00000100:00080000:3.0:1788572597.434069:0:2402:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-MDT0001-mdc-ffff94d402e39800: Resetting ns_connect_flags to server flags: 0xae7a7e7be344d3b8 00000100:00080000:3.0:1788572597.434075:0:2402:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-MDT0001_UUID 00000100:00080000:3.0:1788572597.434078:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to FULL 00000100:00100000:1.0:1788572597.434137:0:70961:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4156bd180 going to sleep for 11 seconds 00000100:00100000:0.0:1788572597.434242:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488543232:192.168.202.125@tcp:38:mount.lustre.0 00000100:00100000:3.0:1788572597.435440:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:86f37580-247f-4c91-a842-1b0c4422007b:0:1875450488543360:192.168.202.125@tcp:41:mount.lustre.0 00000100:00100000:1.0:1788572597.435460:0:70961:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job mount.lustre:86f37580-247f-4c91-a842-1b0c4422007b:70961:1875450488543488:192.168.202.125@tcp:41:mount.lustre.0 00000100:00100000:3.0:1788572597.435461:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:86f37580-247f-4c91-a842-1b0c4422007b:0:1875450488543104:192.168.202.125@tcp:41:mount.lustre.0 00800000:00000004:1.0:1788572597.435482:0:70961:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0000-mdc-ffff94d402e39800: update ffff94d404a1b7d0 cache blocks 317129/353529 objects 1023601/1024000 00000080:00000004:1.0:1788572597.435492:0:70961:0:(obd_class.h:1117:obd_statfs()) lustre-clilmv-ffff94d402e39800: update ffff94d404a18450 cache blocks 317129/353529 objects 1023601/1024000 00000100:00100000:1.0:1788572597.435526:0:70961:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job mount.lustre:86f37580-247f-4c91-a842-1b0c4422007b:70961:1875450488543616:192.168.202.125@tcp:40:mount.lustre.0 00000100:00100000:1.0:1788572597.435572:0:70961:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408caff00 going to sleep for 11 seconds 00000100:00100000:1.0:1788572597.436392:0:70961:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job mount.lustre:86f37580-247f-4c91-a842-1b0c4422007b:70961:1875450488543616:192.168.202.125@tcp:40:mount.lustre.0 00000080:00000004:1.0:1788572597.436467:0:70961:0:(llite_lib.c:622:client_common_fill_super()) rootfid [0x200000007:0x1:0x0] 00000080:00100000:1.0:1788572597.436478:0:70961:0:(llite_lib.c:784:client_common_fill_super()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 2146304 00020000:01000000:1.0:1788572597.436482:0:70961:0:(lov_obd.c:186:lov_connect()) connect #0 00000020:00000080:1.0:1788572597.436492:0:70961:0:(genops.c:1406:class_connect()) connect: client 86f37580-247f-4c91-a842-1b0c4422007b, cookie 0x1e260726fbbc0c34 00000020:00000080:1.0:1788572597.436506:0:70961:0:(genops.c:1406:class_connect()) connect: client LOV_OSC_UUID, cookie 0x1e260726fbbc0c3b 00000100:00080000:1.0:1788572597.436514:0:70961:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 0 00000100:00080000:1.0:1788572597.436521:0:70961:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572597.436531:0:70961:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 134217728, lost: 0. 00000100:00100000:1.0:1788572597.436546:0:70961:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488543744/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572597.436572:0:70961:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID 00020000:01000000:1.0:1788572597.436590:0:70961:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 0 lustre-OST0000_UUID (lustre-OST0000-osc-ffff94d402e39800) active 00000020:00000080:1.0:1788572597.436621:0:70961:0:(genops.c:1406:class_connect()) connect: client LOV_OSC_UUID, cookie 0x1e260726fbbc0c42 00000100:00080000:1.0:1788572597.436626:0:70961:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 0 00000100:00080000:1.0:1788572597.436631:0:70961:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572597.436636:0:70961:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 134217728, lost: 0. 00000100:00100000:1.0:1788572597.436649:0:70961:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd880 x1875450488543872/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572597.436670:0:70961:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID 00020000:01000000:1.0:1788572597.436684:0:70961:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 1 lustre-OST0001_UUID (lustre-OST0001-osc-ffff94d402e39800) active 00000100:00100000:0.0:1788572597.436697:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488543744:192.168.202.125@tcp:8:mount.lustre.0 00000100:00100000:1.0:1788572597.436716:0:70961:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:86f37580-247f-4c91-a842-1b0c4422007b:70961:1875450488544000:192.168.202.125@tcp:900:mount.lustre.0 00000100:00100000:1.0:1788572597.436750:0:70961:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408caff00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572597.436753:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488543872:192.168.202.125@tcp:8:mount.lustre.0 00000100:00100000:0.0:1788572597.437782:0:70961:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:86f37580-247f-4c91-a842-1b0c4422007b:70961:1875450488544000:192.168.202.125@tcp:900:mount.lustre.0 00000100:00100000:0.0:1788572597.437828:0:70961:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3fb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:86f37580-247f-4c91-a842-1b0c4422007b:70961:1875450488544128:192.168.202.125@tcp:33:mount.lustre.0 00000100:00100000:0.0:1788572597.437912:0:70961:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235840 going to sleep for 11 seconds 00000100:00100000:0.0:1788572597.438085:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-OST0000-osc-ffff94d402e39800: committing for last_committed 0 gen 1 00000100:00080000:0.0:1788572597.438093:0:2402:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff94d402e39800: connect to target with instance 1 00000100:00080000:0.0:1788572597.438101:0:2402:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff94d402e39800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:0.0:1788572597.438109:0:2402:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:0.0:1788572597.438112:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00020000:01000000:0.0:1788572597.438118:0:2402:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0000_UUID active 00000080:00000004:0.0:1788572597.438122:0:2402:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0467af2e3640038 -> 0xa0427af2e3440038 00000080:00080000:0.0:1788572597.438125:0:2402:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:0.0:1788572597.438135:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488543744:192.168.202.125@tcp:8:mount.lustre.0 00000100:00100000:0.0:1788572597.438247:0:2402:0:(client.c:3112:ptlrpc_free_committed()) lustre-OST0001-osc-ffff94d402e39800: committing for last_committed 0 gen 1 00000100:00080000:0.0:1788572597.438252:0:2402:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff94d402e39800: connect to target with instance 1 00000100:00080000:0.0:1788572597.438258:0:2402:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff94d402e39800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:0.0:1788572597.438263:0:2402:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:0.0:1788572597.438265:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00020000:01000000:0.0:1788572597.438269:0:2402:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0001_UUID active 00000080:00000004:0.0:1788572597.438271:0:2402:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:0.0:1788572597.438273:0:2402:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:0.0:1788572597.438277:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488543872:192.168.202.125@tcp:8:mount.lustre.0 00000100:00100000:0.0:1788572597.439279:0:70961:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3fb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:86f37580-247f-4c91-a842-1b0c4422007b:70961:1875450488544128:192.168.202.125@tcp:33:mount.lustre.0 00000080:00200000:0.0:1788572597.439447:0:70961:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) 00000080:00200000:0.0:1788572597.439456:0:70961:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:02000400:0.0:1788572597.439513:0:70961:0:(llite_lib.c:1662:ll_fill_super()) Mounted lustre-client - version 2.17.57_1_g3e0b683 00000080:00000004:0.0:1788572597.441444:0:70961:0:(super25.c:182:lustre_fill_super()) 192.168.202.125@tcp:/lustre: Mount complete 00010000:00010000:0.0:1788572598.405357:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:1.0:1788572598.788076:0:71029:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d410bcaa08), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572598.788090:0:71029:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08),name=/ 00000002:00010000:1.0:1788572598.788115:0:71029:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572598.788171:0:71029:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 71029 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.788179:0:71029:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 71029 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.788189:0:71029:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 71029 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572598.788235:0:71029:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job lfs:86f37580-247f-4c91-a842-1b0c4422007b:71029:1875450488544256:192.168.202.125@tcp:101:lfs.0 00000100:00100000:1.0:1788572598.788310:0:71029:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4125fa900 going to sleep for 11 seconds 00000100:00100000:1.0:1788572598.791592:0:71029:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job lfs:86f37580-247f-4c91-a842-1b0c4422007b:71029:1875450488544256:192.168.202.125@tcp:101:lfs.0 00010000:00010000:1.0:1788572598.791627:0:71029:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e101a expref: -99 pid: 71029 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.791648:0:71029:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e101a expref: -99 pid: 71029 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.791662:0:71029:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e101a expref: -99 pid: 71029 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.791673:0:71029:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e101a expref: -99 pid: 71029 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.791686:0:71029:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e101a expref: -99 pid: 71029 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.791694:0:71029:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e101a expref: -99 pid: 71029 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.791702:0:71029:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e101a expref: -99 pid: 71029 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572598.791718:0:71029:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff94d4049be680 x1875450488544256/t0(0) o101->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572598.791732:0:71029:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e101a expref: -99 pid: 71029 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.791744:0:71029:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d405c86c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788572598.791751:0:71029:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572598.791787:0:71029:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1788572598.791792:0:71029:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d410bcaa08) for lock 0x1e260726fbbc0c49 00000080:00010000:1.0:1788572598.791797:0:71029:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d410bcaa08) 00000080:00010000:1.0:1788572598.791800:0:71029:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0c49 from it ffffb68a4235fcb8 00010000:00010000:1.0:1788572598.791804:0:71029:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e101a expref: -99 pid: 71029 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.791812:0:71029:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e101a expref: -99 pid: 71029 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:1.0:1788572598.792406:0:71029:0:(file.c:6285:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff94d410bcaa08), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1788572598.792411:0:71029:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08),name=/ 00000002:00010000:1.0:1788572598.792425:0:71029:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572598.792431:0:71029:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e101a expref: -99 pid: 71029 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572598.792439:0:71029:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e101a expref: -99 pid: 71029 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572598.792447:0:71029:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572598.792451:0:71029:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d410bcaa08) 00000080:00010000:1.0:1788572598.792453:0:71029:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d410bcaa08) for lock 0x1e260726fbbc0c49 00000080:00010000:1.0:1788572598.792456:0:71029:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0c49 from it ffffb68a4235fcb8 00010000:00010000:1.0:1788572598.792458:0:71029:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e101a expref: -99 pid: 71029 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572598.792462:0:71029:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e101a expref: -99 pid: 71029 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000080:00200000:1.0:1788572598.792547:0:71029:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08),name=/ 00000002:00010000:1.0:1788572598.792555:0:71029:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572598.792562:0:71029:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e101a expref: -99 pid: 71029 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788572598.792568:0:71029:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e101a expref: -99 pid: 71029 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000080:00200000:1.0:1788572598.792572:0:71029:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572598.792575:0:71029:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d410bcaa08) 00000080:00010000:1.0:1788572598.792577:0:71029:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d410bcaa08) for lock 0x1e260726fbbc0c49 00000080:00010000:1.0:1788572598.792579:0:71029:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0c49 from it ffffb68a4235fbb0 00010000:00010000:1.0:1788572598.792582:0:71029:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e101a expref: -99 pid: 71029 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788572598.792590:0:71029:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e101a expref: -99 pid: 71029 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:1.0:1788572598.792598:0:71029:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08), inode mode 41ed mask 44 00000080:00200008:1.0:1788572598.792619:0:71029:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff94d410bcaa08), flags 100000 00000080:00200080:1.0:1788572598.792706:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.792719:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00800000:00000004:1.0:1788572598.792726:0:71029:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0000-mdc-ffff94d402e39800: age 3750, max_age 3751 00000100:00100000:1.0:1788572598.792755:0:71029:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job lfs:86f37580-247f-4c91-a842-1b0c4422007b:71029:1875450488544384:192.168.202.125@tcp:41:lfs.0 00000100:00100000:1.0:1788572598.792821:0:71029:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4125fa900 going to sleep for 11 seconds 00000100:00100000:1.0:1788572598.793951:0:71029:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job lfs:86f37580-247f-4c91-a842-1b0c4422007b:71029:1875450488544384:192.168.202.125@tcp:41:lfs.0 00800000:00000004:1.0:1788572598.793967:0:71029:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0000-mdc-ffff94d402e39800: update ffff94d404a1b7d0 cache blocks 317129/353529 objects 1023601/1024000 00000080:00200080:1.0:1788572598.794016:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.794025:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00800000:00000004:1.0:1788572598.794029:0:71029:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0001-mdc-ffff94d402e39800: age 2750, max_age 3751 00000100:00100000:1.0:1788572598.794043:0:71029:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job lfs:86f37580-247f-4c91-a842-1b0c4422007b:71029:1875450488544512:192.168.202.125@tcp:41:lfs.0 00000100:00100000:1.0:1788572598.794085:0:71029:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4125fa900 going to sleep for 11 seconds 00000100:00100000:1.0:1788572598.795006:0:71029:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job lfs:86f37580-247f-4c91-a842-1b0c4422007b:71029:1875450488544512:192.168.202.125@tcp:41:lfs.0 00800000:00000004:1.0:1788572598.795022:0:71029:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0001-mdc-ffff94d402e39800: update ffff94d4113f5190 cache blocks 317205/353529 objects 1023682/1024000 00000080:00200080:1.0:1788572598.795052:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795060:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795065:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795070:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795074:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795078:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795085:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795089:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795093:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795098:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795103:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795107:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795111:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795116:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795121:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795126:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795130:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795148:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795154:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795160:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795166:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795170:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795175:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795179:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795184:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795188:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795193:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795197:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795222:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795228:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795233:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795237:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795241:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795250:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795258:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795264:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795269:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795273:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795277:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795281:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795284:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795288:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795293:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795297:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795302:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795306:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795311:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795314:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795318:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795321:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795325:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795328:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795332:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795335:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795339:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795342:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795346:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795349:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795353:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795357:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795361:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795365:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795369:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795373:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795377:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795380:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795385:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795388:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795393:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795396:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795400:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795403:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795407:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795410:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795414:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795422:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795427:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795431:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795435:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795438:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795442:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795446:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795449:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795452:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795456:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795459:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795462:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795465:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795468:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795471:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795474:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795477:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795480:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795483:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795487:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795489:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795493:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795496:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795499:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795502:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795505:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795508:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795511:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795514:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795518:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795522:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795526:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795529:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795533:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795536:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795540:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795544:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795548:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795551:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795554:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795556:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795560:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795563:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795567:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795571:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795575:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795578:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795583:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795586:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795590:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00800000:00000080:1.0:1788572598.795593:0:71029:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.795597:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.795619:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00020000:00000004:1.0:1788572598.795627:0:71029:0:(obd_class.h:1091:obd_statfs()) lustre-OST0000-osc-ffff94d402e39800: age 2750, max_age 3751 00000100:00100000:1.0:1788572598.795650:0:71029:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job lfs:86f37580-247f-4c91-a842-1b0c4422007b:71029:1875450488544640:192.168.202.125@tcp:13:lfs.0 00000100:00100000:1.0:1788572598.795746:0:71029:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4125fa900 going to sleep for 11 seconds 00000100:00100000:1.0:1788572598.797339:0:71029:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job lfs:86f37580-247f-4c91-a842-1b0c4422007b:71029:1875450488544640:192.168.202.125@tcp:13:lfs.0 00020000:00000004:1.0:1788572598.797360:0:71029:0:(obd_class.h:1117:obd_statfs()) lustre-OST0000-osc-ffff94d402e39800: update ffff94d4113f1e10 cache blocks 901369/958279 objects 261731/262144 00000080:00200080:1.0:1788572598.797412:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.797424:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00020000:00000004:1.0:1788572598.797429:0:71029:0:(obd_class.h:1091:obd_statfs()) lustre-OST0001-osc-ffff94d402e39800: age 2750, max_age 3751 00000100:00100000:1.0:1788572598.797452:0:71029:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job lfs:86f37580-247f-4c91-a842-1b0c4422007b:71029:1875450488544768:192.168.202.125@tcp:13:lfs.0 00000100:00100000:1.0:1788572598.797516:0:71029:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d4125fa900 going to sleep for 11 seconds 00000100:00100000:1.0:1788572598.798646:0:71029:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job lfs:86f37580-247f-4c91-a842-1b0c4422007b:71029:1875450488544768:192.168.202.125@tcp:13:lfs.0 00020000:00000004:1.0:1788572598.798669:0:71029:0:(obd_class.h:1117:obd_statfs()) lustre-OST0001-osc-ffff94d402e39800: update ffff94d4113f37d0 cache blocks 901113/958279 objects 261732/262144 00000080:00200080:1.0:1788572598.798739:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798747:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798752:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798754:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798757:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798759:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798761:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798763:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798765:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798767:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798769:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798771:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798773:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798774:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798776:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798779:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798781:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798783:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798785:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798787:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798789:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798791:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798793:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798794:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798796:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798798:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798800:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798801:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798803:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798805:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798807:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798808:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798810:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798812:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798814:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798815:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798817:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798819:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798821:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798822:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798825:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798826:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798828:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798830:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798832:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798833:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798835:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798837:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798839:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798841:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798842:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798844:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798846:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798848:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798849:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798851:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798853:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798855:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798857:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798858:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798860:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798862:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798864:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798865:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798867:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798869:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798871:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798873:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798876:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798879:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798883:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798887:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798890:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798893:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798897:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798900:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798904:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798906:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798910:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798913:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798917:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798920:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798924:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798927:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798931:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798934:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798937:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798941:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798945:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798948:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798952:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798956:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798959:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798962:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798966:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798970:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798974:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798978:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798982:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798985:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798989:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798992:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.798996:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.798999:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.799003:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.799006:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.799010:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.799014:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.799018:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.799021:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.799025:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.799028:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.799032:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.799036:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.799039:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.799043:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.799047:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.799050:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.799054:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.799058:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.799061:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.799065:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.799069:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.799072:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200080:1.0:1788572598.799076:0:71029:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08) cmd=c00866a4 arg=7ffea6e87f90 00020000:00000080:1.0:1788572598.799079:0:71029:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff94d402e39800: cmd=c00866a4 len=592 karg=ffff94d416e24400 uarg=0000000000000000 00000080:00200008:1.0:1788572598.799093:0:71029:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff94d410bcaa08), flags 100000 00000080:00200000:3.0:1788572599.133749:0:71063:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08),name=/ 00000002:00010000:3.0:1788572599.133781:0:71063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572599.133802:0:71063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e101a expref: -99 pid: 71029 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788572599.133819:0:71063:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e101a expref: -99 pid: 71029 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:3.0:1788572599.133834:0:71063:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572599.133841:0:71063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d410bcaa08) 00000080:00010000:3.0:1788572599.133845:0:71063:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d410bcaa08) for lock 0x1e260726fbbc0c49 00000080:00010000:3.0:1788572599.133852:0:71063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0c49 from it ffffb68a41533b58 00010000:00010000:3.0:1788572599.133856:0:71063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e101a expref: -99 pid: 71029 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788572599.133865:0:71063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e101a expref: -99 pid: 71029 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:3.0:1788572599.133879:0:71063:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572599.133893:0:71063:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d133.replay-single, dir=[0x200000007:0x1:0x0](ffff94d410bcaa08), flags=25 00000080:00200000:3.0:1788572599.133898:0:71063:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d133.replay-single, dir=[0x200000007:0x1:0x0](ffff94d410bcaa08), intent=0 00000002:00010000:3.0:1788572599.133910:0:71063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d133.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1788572599.133952:0:71063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d408cf1e00/0x1e260726fbbc0c50 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572599.133960:0:71063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d408cf1e00/0x1e260726fbbc0c50 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572599.133969:0:71063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d408cf1e00/0x1e260726fbbc0c50 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572599.133992:0:71063:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job cp:86f37580-247f-4c91-a842-1b0c4422007b:71063:1875450488544896:192.168.202.125@tcp:101:cp.0 00000100:00100000:3.0:1788572599.134082:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d41207fa80 going to sleep for 11 seconds 00000100:00100000:0.0:1788572599.135867:0:71063:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job cp:86f37580-247f-4c91-a842-1b0c4422007b:71063:1875450488544896:192.168.202.125@tcp:101:cp.0 00010000:00010000:0.0:1788572599.135895:0:71063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d408cf1e00/0x1e260726fbbc0c50 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e1036 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.135911:0:71063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d408cf1e00/0x1e260726fbbc0c50 lrc: 4/1,0 mode: --/PR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e1036 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.135917:0:71063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d408cf1e00/0x1e260726fbbc0c50 lrc: 4/1,0 mode: --/PR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e1036 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.135924:0:71063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d408cf1e00/0x1e260726fbbc0c50 lrc: 4/1,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e1036 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.135932:0:71063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d408cf1e00/0x1e260726fbbc0c50 lrc: 4/1,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e1036 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.135942:0:71063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d408cf1e00/0x1e260726fbbc0c50 lrc: 5/2,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e1036 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.135948:0:71063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d408cf1e00/0x1e260726fbbc0c50 lrc: 5/2,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e1036 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.135956:0:71063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d408cf1e00/0x1e260726fbbc0c50 lrc: 4/1,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e1036 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572599.135966:0:71063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff94d408c54380 x1875450488544896/t0(0) o101->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 592/584 e 0 to 0 dl 1788572615 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572599.135977:0:71063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d408cf1e00/0x1e260726fbbc0c50 lrc: 3/1,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e1036 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.135986:0:71063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d405c86c00 type 13 mode 4 res 9663703441/12 (0 0) 00000100:00100000:0.0:1788572599.136022:0:71063:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3fb80 pname:cluuid:pid:xid:nid:opc:job cp:86f37580-247f-4c91-a842-1b0c4422007b:71063:1875450488545024:192.168.202.125@tcp:900:cp.0 00000100:00100000:0.0:1788572599.136072:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 11 seconds 00000100:00100000:0.0:1788572599.136946:0:71063:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3fb80 pname:cluuid:pid:xid:nid:opc:job cp:86f37580-247f-4c91-a842-1b0c4422007b:71063:1875450488545024:192.168.202.125@tcp:900:cp.0 00000002:00010000:0.0:1788572599.136969:0:71063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240006991:0xc:0x0], intent: lookup flags 0 00010000:00010000:0.0:1788572599.136994:0:71063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412167200/0x1e260726fbbc0c57 lrc: 3/1,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.137000:0:71063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412167200/0x1e260726fbbc0c57 lrc: 3/1,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.137006:0:71063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412167200/0x1e260726fbbc0c57 lrc: 3/1,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572599.137013:0:71063:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3fb80 pname:cluuid:pid:xid:nid:opc:job cp:86f37580-247f-4c91-a842-1b0c4422007b:71063:1875450488545152:192.168.202.125@tcp:101:cp.0 00000100:00100000:0.0:1788572599.137041:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 11 seconds 00000100:00100000:0.0:1788572599.141717:0:71063:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3fb80 pname:cluuid:pid:xid:nid:opc:job cp:86f37580-247f-4c91-a842-1b0c4422007b:71063:1875450488545152:192.168.202.125@tcp:101:cp.0 00010000:00010000:0.0:1788572599.141742:0:71063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412167200/0x1e260726fbbc0c57 lrc: 4/1,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e1044 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.141755:0:71063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412167200/0x1e260726fbbc0c57 lrc: 4/1,0 mode: --/PR res: [0x240006991:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e1044 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.141766:0:71063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412167200/0x1e260726fbbc0c57 lrc: 4/1,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e1044 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.141781:0:71063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412167200/0x1e260726fbbc0c57 lrc: 4/1,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e1044 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.141799:0:71063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412167200/0x1e260726fbbc0c57 lrc: 5/2,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e1044 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.141805:0:71063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412167200/0x1e260726fbbc0c57 lrc: 5/2,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e1044 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.141810:0:71063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412167200/0x1e260726fbbc0c57 lrc: 4/1,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e1044 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572599.141817:0:71063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff94d408d3fb80 x1875450488545152/t0(0) o101->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 576/584 e 0 to 0 dl 1788572615 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572599.141826:0:71063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412167200/0x1e260726fbbc0c57 lrc: 3/1,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e1044 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.141834:0:71063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d408d3a400 type 13 mode 4 res 9663703441/12 (0 0) 00000080:00200000:0.0:1788572599.141850:0:71063:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788572599.141880:0:71063:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x240006991:0xc:0x0](ffff94d410bcd348) 00000080:00200000:0.0:1788572599.141888:0:71063:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x240006991:0xc:0x0], updating i_size 4096 00000080:00010000:0.0:1788572599.141894:0:71063:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x240006991:0xc:0x0](ffff94d410bcd348) for remote lock 0x1e260726fbbc0c57 00000080:00010000:0.0:1788572599.141898:0:71063:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240006991:0xc:0x0](ffff94d410bcd348) for lock 0x1e260726fbbc0c50 00000080:00010000:0.0:1788572599.141906:0:71063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240006991:0xc:0x0](ffff94d410bcd348) 00000080:00010000:0.0:1788572599.141908:0:71063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0c50 from it ffffb68a41533a28 00010000:00010000:0.0:1788572599.141911:0:71063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d408cf1e00/0x1e260726fbbc0c50 lrc: 3/1,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e1036 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.141918:0:71063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d408cf1e00/0x1e260726fbbc0c50 lrc: 2/0,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e1036 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788572599.141928:0:71063:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x1e260726fbbc0c57 from it ffffb68a41533a28 00010000:00010000:0.0:1788572599.141930:0:71063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412167200/0x1e260726fbbc0c57 lrc: 3/1,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e1044 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.141935:0:71063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412167200/0x1e260726fbbc0c57 lrc: 2/0,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e1044 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572599.141950:0:71063:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x240006991:0xc:0x0](ffff94d410bcd348), inode mode 41ed mask 1 00000080:00200000:0.0:1788572599.141965:0:71063:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=remote_dir, dir=[0x240006991:0xc:0x0](ffff94d410bcd348), flags=25 00000080:00200000:0.0:1788572599.141968:0:71063:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=remote_dir, dir=[0x240006991:0xc:0x0](ffff94d410bcd348), intent=0 00000002:00010000:0.0:1788572599.141982:0:71063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: remote_dir,[0x0:0x0:0x0]) in obj [0x240006991:0xc:0x0], intent: lookup flags 0 00010000:00010000:0.0:1788572599.142011:0:71063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412166800/0x1e260726fbbc0c5e lrc: 3/1,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.142016:0:71063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412166800/0x1e260726fbbc0c5e lrc: 3/1,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.142022:0:71063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412166800/0x1e260726fbbc0c5e lrc: 3/1,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572599.142037:0:71063:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3fb80 pname:cluuid:pid:xid:nid:opc:job cp:86f37580-247f-4c91-a842-1b0c4422007b:71063:1875450488545280:192.168.202.125@tcp:101:cp.0 00000100:00100000:0.0:1788572599.142098:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 11 seconds 00000100:00100000:0.0:1788572599.143772:0:71063:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3fb80 pname:cluuid:pid:xid:nid:opc:job cp:86f37580-247f-4c91-a842-1b0c4422007b:71063:1875450488545280:192.168.202.125@tcp:101:cp.0 00010000:00010000:0.0:1788572599.143792:0:71063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412166800/0x1e260726fbbc0c5e lrc: 4/1,0 mode: --/CR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e1060 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.143810:0:71063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412166800/0x1e260726fbbc0c5e lrc: 4/1,0 mode: --/PR res: [0x240006991:0xd:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e1060 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.143818:0:71063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412166800/0x1e260726fbbc0c5e lrc: 4/1,0 mode: --/PR res: [0x240006991:0xd:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e1060 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.143828:0:71063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412166800/0x1e260726fbbc0c5e lrc: 4/1,0 mode: PR/PR res: [0x240006991:0xd:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e1060 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.143837:0:71063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412166800/0x1e260726fbbc0c5e lrc: 4/1,0 mode: PR/PR res: [0x240006991:0xd:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e1060 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.143850:0:71063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412166800/0x1e260726fbbc0c5e lrc: 5/2,0 mode: PR/PR res: [0x240006991:0xd:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e1060 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.143858:0:71063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412166800/0x1e260726fbbc0c5e lrc: 5/2,0 mode: PR/PR res: [0x240006991:0xd:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e1060 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.143865:0:71063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412166800/0x1e260726fbbc0c5e lrc: 4/1,0 mode: PR/PR res: [0x240006991:0xd:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e1060 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572599.143876:0:71063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff94d408d3fb80 x1875450488545280/t0(0) o101->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 584/688 e 0 to 0 dl 1788572615 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572599.143888:0:71063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412166800/0x1e260726fbbc0c5e lrc: 3/1,0 mode: PR/PR res: [0x240006991:0xd:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e1060 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.143896:0:71063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d408d3a400 type 13 mode 4 res 9663703441/13 (0 0) 00000080:00200000:0.0:1788572599.143901:0:71063:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x240006991:0xc:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788572599.143925:0:71063:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x240006991:0xd:0x0](ffff94d410bc9148) 00000080:00200000:0.0:1788572599.143940:0:71063:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x240006991:0xd:0x0], updating i_size 4096 00000080:00010000:0.0:1788572599.143946:0:71063:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240006991:0xd:0x0](ffff94d410bc9148) for lock 0x1e260726fbbc0c5e 00000080:00010000:0.0:1788572599.143957:0:71063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240006991:0xd:0x0](ffff94d410bc9148) 00000080:00010000:0.0:1788572599.143960:0:71063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0c5e from it ffffb68a41533a28 00010000:00010000:0.0:1788572599.143963:0:71063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412166800/0x1e260726fbbc0c5e lrc: 3/1,0 mode: PR/PR res: [0x240006991:0xd:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e1060 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.143970:0:71063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412166800/0x1e260726fbbc0c5e lrc: 2/0,0 mode: PR/PR res: [0x240006991:0xd:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e1060 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572599.143995:0:71063:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x240006991:0xd:0x0](ffff94d410bc9148), inode mode 41ed mask 1 00000080:00200000:0.0:1788572599.144007:0:71063:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file, dir=[0x240006991:0xd:0x0](ffff94d410bc9148), flags=9 00000080:00200000:0.0:1788572599.144011:0:71063:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file, dir=[0x240006991:0xd:0x0](ffff94d410bc9148), intent=getattr 00000002:00010000:0.0:1788572599.144025:0:71063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file,[0x0:0x0:0x0]) in obj [0x240006991:0xd:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788572599.144053:0:71063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412166c00/0x1e260726fbbc0c65 lrc: 3/1,0 mode: --/CR res: [0x240006991:0xd:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.144061:0:71063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412166c00/0x1e260726fbbc0c65 lrc: 3/1,0 mode: --/CR res: [0x240006991:0xd:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.144069:0:71063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412166c00/0x1e260726fbbc0c65 lrc: 3/1,0 mode: --/CR res: [0x240006991:0xd:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572599.144084:0:71063:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3fb80 pname:cluuid:pid:xid:nid:opc:job cp:86f37580-247f-4c91-a842-1b0c4422007b:71063:1875450488545408:192.168.202.125@tcp:101:cp.0 00000100:00100000:0.0:1788572599.144139:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 11 seconds 00000100:00100000:0.0:1788572599.145493:0:71063:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3fb80 pname:cluuid:pid:xid:nid:opc:job cp:86f37580-247f-4c91-a842-1b0c4422007b:71063:1875450488545408:192.168.202.125@tcp:101:cp.0 00010000:00010000:0.0:1788572599.145506:0:71063:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412166c00/0x1e260726fbbc0c65 lrc: 4/1,0 mode: --/CR res: [0x240006991:0xd:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.145519:0:71063:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412166c00/0x1e260726fbbc0c65 lrc: 4/1,0 mode: --/CR res: [0x240006991:0xd:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.145524:0:71063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412166c00/0x1e260726fbbc0c65 lrc: 4/1,0 mode: --/CR res: [0x240006991:0xd:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.145530:0:71063:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412166c00/0x1e260726fbbc0c65 lrc: 3/0,0 mode: --/CR res: [0x240006991:0xd:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.145536:0:71063:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412166c00/0x1e260726fbbc0c65 lrc: 4/0,0 mode: --/CR res: [0x240006991:0xd:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.145542:0:71063:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412166c00 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:0.0:1788572599.145546:0:71063:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412166c00/0x1e260726fbbc0c65 lrc: 4/0,0 mode: --/CR res: [0x240006991:0xd:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.145551:0:71063:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412166c00/0x1e260726fbbc0c65 lrc: 4/0,0 mode: --/CR res: [0x240006991:0xd:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.145557:0:71063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412166c00/0x1e260726fbbc0c65 lrc: 5/0,0 mode: --/CR res: [0x240006991:0xd:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.145563:0:71063:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788572599.145568:0:71063:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412166c00/0x1e260726fbbc0c65 lrc: 3/0,0 mode: --/CR res: [0x240006991:0xd:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.145573:0:71063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412166c00/0x1e260726fbbc0c65 lrc: 0/0,0 mode: --/CR res: [0x240006991:0xd:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572599.145584:0:71063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff94d408d3fb80 x1875450488545408/t0(0) o101->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 576/584 e 0 to 0 dl 1788572615 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572599.145612:0:71063:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240006991:0xd:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1788572599.145630:0:71063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d408d3a400 type 13 mode 30 res 9663703441/13 (0 0) 00000080:00200000:0.0:1788572599.145691:0:71063:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08),name=/ 00000002:00010000:0.0:1788572599.145703:0:71063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572599.145715:0:71063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e101a expref: -99 pid: 71029 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:0.0:1788572599.145726:0:71063:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e101a expref: -99 pid: 71029 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:0.0:1788572599.145734:0:71063:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572599.145739:0:71063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d410bcaa08) 00000080:00010000:0.0:1788572599.145742:0:71063:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d410bcaa08) for lock 0x1e260726fbbc0c49 00000080:00010000:0.0:1788572599.145746:0:71063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0c49 from it ffffb68a41533b58 00010000:00010000:0.0:1788572599.145750:0:71063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e101a expref: -99 pid: 71029 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:0.0:1788572599.145757:0:71063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e101a expref: -99 pid: 71029 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:0.0:1788572599.145769:0:71063:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572599.145775:0:71063:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d133.replay-single, flags=25 00000080:00200000:0.0:1788572599.145781:0:71063:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x240006991:0xc:0x0](ffff94d410bcd348), inode mode 41ed mask 1 00000080:00200000:0.0:1788572599.145784:0:71063:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:0.0:1788572599.145786:0:71063:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x240006991:0xd:0x0](ffff94d410bc9148), inode mode 41ed mask 1 00000080:00200000:0.0:1788572599.145793:0:71063:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file, dir=[0x240006991:0xd:0x0](ffff94d410bc9148), flags=9 00000080:00200000:0.0:1788572599.145796:0:71063:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file, dir=[0x240006991:0xd:0x0](ffff94d410bc9148), intent=getattr 00000002:00010000:0.0:1788572599.145806:0:71063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file,[0x0:0x0:0x0]) in obj [0x240006991:0xd:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788572599.145832:0:71063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412167800/0x1e260726fbbc0c6c lrc: 3/1,0 mode: --/CR res: [0x240006991:0xd:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.145840:0:71063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412167800/0x1e260726fbbc0c6c lrc: 3/1,0 mode: --/CR res: [0x240006991:0xd:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.145847:0:71063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412167800/0x1e260726fbbc0c6c lrc: 3/1,0 mode: --/CR res: [0x240006991:0xd:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572599.145862:0:71063:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3fb80 pname:cluuid:pid:xid:nid:opc:job cp:86f37580-247f-4c91-a842-1b0c4422007b:71063:1875450488545536:192.168.202.125@tcp:101:cp.0 00000100:00100000:0.0:1788572599.145916:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 11 seconds 00000100:00100000:0.0:1788572599.150001:0:71063:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3fb80 pname:cluuid:pid:xid:nid:opc:job cp:86f37580-247f-4c91-a842-1b0c4422007b:71063:1875450488545536:192.168.202.125@tcp:101:cp.0 00010000:00010000:0.0:1788572599.150018:0:71063:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412167800/0x1e260726fbbc0c6c lrc: 4/1,0 mode: --/CR res: [0x240006991:0xd:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.150027:0:71063:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412167800/0x1e260726fbbc0c6c lrc: 4/1,0 mode: --/CR res: [0x240006991:0xd:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.150032:0:71063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412167800/0x1e260726fbbc0c6c lrc: 4/1,0 mode: --/CR res: [0x240006991:0xd:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.150036:0:71063:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412167800/0x1e260726fbbc0c6c lrc: 3/0,0 mode: --/CR res: [0x240006991:0xd:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.150042:0:71063:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412167800/0x1e260726fbbc0c6c lrc: 4/0,0 mode: --/CR res: [0x240006991:0xd:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.150046:0:71063:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff94d412167800 already unused, calling callback (ffffffffc19072d0) 00010000:00010000:0.0:1788572599.150049:0:71063:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412167800/0x1e260726fbbc0c6c lrc: 4/0,0 mode: --/CR res: [0x240006991:0xd:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.150054:0:71063:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412167800/0x1e260726fbbc0c6c lrc: 4/0,0 mode: --/CR res: [0x240006991:0xd:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.150059:0:71063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412167800/0x1e260726fbbc0c6c lrc: 5/0,0 mode: --/CR res: [0x240006991:0xd:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.150062:0:71063:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788572599.150067:0:71063:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412167800/0x1e260726fbbc0c6c lrc: 3/0,0 mode: --/CR res: [0x240006991:0xd:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572599.150070:0:71063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412167800/0x1e260726fbbc0c6c lrc: 0/0,0 mode: --/CR res: [0x240006991:0xd:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 71063 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572599.150082:0:71063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff94d408d3fb80 x1875450488545536/t0(0) o101->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 576/584 e 0 to 0 dl 1788572615 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572599.150093:0:71063:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240006991:0xd:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1788572599.150113:0:71063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff94d408d3a400 type 13 mode 30 res 9663703441/13 (0 0) 00000080:00200000:0.0:1788572599.150267:0:71063:0:(file.c:6180:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08),name=/ 00000002:00010000:0.0:1788572599.150282:0:71063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572599.150293:0:71063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e101a expref: -99 pid: 71029 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:0.0:1788572599.150303:0:71063:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e101a expref: -99 pid: 71029 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:0.0:1788572599.150310:0:71063:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572599.150315:0:71063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d410bcaa08) 00000080:00010000:0.0:1788572599.150318:0:71063:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff94d410bcaa08) for lock 0x1e260726fbbc0c49 00000080:00010000:0.0:1788572599.150323:0:71063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1e260726fbbc0c49 from it ffffb68a41533bf8 00010000:00010000:0.0:1788572599.150327:0:71063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xec901abfeb4e101a expref: -99 pid: 71029 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:0.0:1788572599.150336:0:71063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d412583e00/0x1e260726fbbc0c49 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xec901abfeb4e101a expref: -99 pid: 71029 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:0.0:1788572599.150351:0:71063:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff94d410bcaa08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572599.150357:0:71063:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d133.replay-single, flags=25 00000080:00200000:0.0:1788572599.150361:0:71063:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x240006991:0xc:0x0](ffff94d410bcd348), inode mode 41ed mask 1 00000080:00200000:0.0:1788572599.150363:0:71063:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:0.0:1788572599.150366:0:71063:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x240006991:0xd:0x0](ffff94d410bc9148), inode mode 41ed mask 1 00000080:00200000:0.0:1788572599.150377:0:71063:0:(file.c:6661:ll_inode_permission()) VFS Op:inode=[0x240006991:0xd:0x0](ffff94d410bc9148), inode mode 41ed mask 3 00000080:00200000:0.0:1788572599.150434:0:71063:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file, dir=[0x240006991:0xd:0x0](ffff94d410bc9148), file ffff94d4039b4000, open_flags 80c1, mode 81a4 opened 0 00000080:00200000:0.0:1788572599.150443:0:71063:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file, dir=[0x240006991:0xd:0x0](ffff94d410bc9148), intent=open|creat 00000100:00100000:0.0:1788572599.150486:0:71063:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3fb80 pname:cluuid:pid:xid:nid:opc:job cp:86f37580-247f-4c91-a842-1b0c4422007b:71063:1875450488545664:192.168.202.125@tcp:700:cp.0 00000100:00100000:0.0:1788572599.150539:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 11 seconds 00010000:00010000:1.0:1788572599.429360:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572600.453377:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572601.477395:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572602.501429:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572602.565520:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2403:1875450488545792:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572602.565617:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:86f37580-247f-4c91-a842-1b0c4422007b:2404:1875450488545920:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572602.565650:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:86f37580-247f-4c91-a842-1b0c4422007b:2403:1875450488546048:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572602.565670:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:86f37580-247f-4c91-a842-1b0c4422007b:2404:1875450488546176:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572602.565681:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:86f37580-247f-4c91-a842-1b0c4422007b:2403:1875450488546304:192.168.202.125@tcp:400:kworker.0 00010000:00010000:1.0:1788572603.525377:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572604.549433:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572605.573333:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572606.597393:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572607.621355:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572607.685401:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2404:1875450488546432:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572607.685408:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:86f37580-247f-4c91-a842-1b0c4422007b:2403:1875450488546560:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572607.685494:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:86f37580-247f-4c91-a842-1b0c4422007b:2404:1875450488546688:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572607.685496:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:86f37580-247f-4c91-a842-1b0c4422007b:2403:1875450488546816:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572607.685540:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:86f37580-247f-4c91-a842-1b0c4422007b:2404:1875450488546944:192.168.202.125@tcp:400:kworker.0 00010000:00010000:0.0:1788572608.645370:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572609.669334:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572610.245288:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 1 seconds 00010000:00010000:0.0:1788572610.693363:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572611.269269:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 1 seconds 00010000:00010000:1.0:1788572611.717370:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572612.293311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 1 seconds 00010000:00010000:0.0:1788572612.741366:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572612.805404:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2403:1875450488547072:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572612.805419:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:86f37580-247f-4c91-a842-1b0c4422007b:2404:1875450488547200:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572612.805488:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:86f37580-247f-4c91-a842-1b0c4422007b:2404:1875450488547456:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572612.805502:0:2403:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:86f37580-247f-4c91-a842-1b0c4422007b:2403:1875450488547328:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572612.805520:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1788572612.805528:0:2404:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:86f37580-247f-4c91-a842-1b0c4422007b:2404:1875450488547584:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572613.317306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 1 seconds 00010000:00010000:1.0:1788572613.765339:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572614.341281:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 1 seconds 00010000:00010000:0.0:1788572614.789364:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1788572615.365327:0:71063:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572599/real 1788572599] req@ffff94d408d3fb80 x1875450488545664/t0(0) o700->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:30/10 lens 264/248 e 0 to 1 dl 1788572615 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572615.365359:0:71063:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from FULL to DISCONN 00000100:02000400:0.0:1788572615.365364:0:71063:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0001-mdc-ffff94d402e39800: Connection to lustre-MDT0001 (at 192.168.202.125@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:0.0:1788572615.365370:0:71063:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0001_UUID: waking up pinger s:DISCONN 00000100:00100000:0.0:1788572615.365392:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00000100:00080000:2.0:1788572615.365473:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572615.365476:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572615.365479:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572615.365484:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3750 00000100:00080000:2.0:1788572615.365494:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572615.365501:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572615.365514:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488547968/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572615.365517:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2405:1875450488547712:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572615.365582:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488547968:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788572615.365582:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:86f37580-247f-4c91-a842-1b0c4422007b:2406:1875450488547840:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572615.365605:0:2405:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:86f37580-247f-4c91-a842-1b0c4422007b:2405:1875450488548096:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572615.365646:0:2406:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:86f37580-247f-4c91-a842-1b0c4422007b:2406:1875450488548224:192.168.202.125@tcp:400:kworker.0 00010000:00010000:1.0:1788572615.813348:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572616.389306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572616.837352:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572617.413295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572617.861314:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572618.437289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572618.885354:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1788572618.949318:0:2404:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572602/real 1788572602] req@ffff94d408d3ca80 x1875450488545920/t0(0) o400->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572618 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:1.0:1788572618.949318:0:2403:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572602/real 1788572602] req@ffff94d408d3e300 x1875450488545792/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572618 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572618.949370:0:2403:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff94d408d3e300 x1875450488545792/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572618 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572618.949371:0:2404:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff94d408d3ca80 x1875450488545920/t0(0) o400->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572618 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572618.949401:0:2404:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:00080000:1.0:1788572618.949401:0:2403:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from FULL to DISCONN 00000100:02000400:0.0:1788572618.949405:0:2404:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff94d402e39800: Connection to lustre-MDT0000 (at 192.168.202.125@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:02020000:1.0:1788572618.949410:0:2403:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.202.125@tcp: Connection to MGS (at 192.168.202.125@tcp) was lost; in progress operations using this service will fail 00000100:00080000:0.0:1788572618.949411:0:2404:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0000_UUID: waking up pinger s:DISCONN 00000100:00080000:1.0:1788572618.949417:0:2403:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.202.125@tcp for MGC192.168.202.125@tcp not replayable, auto-deactivating 00000100:00080000:1.0:1788572618.949420:0:2403:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:1.0:1788572618.949426:0:2403:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff94d408d3e300 x1875450488545792/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572618 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:0.0:1788572618.949434:0:2404:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572602/real 1788572602] req@ffff94d408d3f100 x1875450488546176/t0(0) o400->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572618 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572618.949441:0:2403:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff94d408d3f800 x1875450488546432/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572623 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572618.949451:0:2404:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff94d408d3f100 x1875450488546176/t0(0) o400->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572618 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572618.949458:0:2403:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff94d408d3d880 x1875450488547072/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572628 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572618.949462:0:2404:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:0.0:1788572618.949472:0:2404:0:(import.c:182:ptlrpc_set_import_discon()) lustre-OST0000-osc-ffff94d402e39800: Connection to lustre-OST0000 (at 192.168.202.125@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00100000:1.0:1788572618.949478:0:2403:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff94d408c60a80 x1875450488547712/t0(0) o400->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572631 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572618.949479:0:2404:0:(import.c:418:ptlrpc_pinger_force()) lustre-OST0000_UUID: waking up pinger s:DISCONN 10000000:01000000:1.0:1788572618.949495:0:2403:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:2.0:1788572618.949497:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572618.949498:0:2403:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:1.0:1788572618.949501:0:2403:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:1.0:1788572618.949506:0:2403:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:2.0:1788572618.949506:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572618.949511:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788572618.949514:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2404:1875450488546432:192.168.202.125@tcp:400:kworker.0 00000100:00000400:1.0:1788572618.949520:0:2403:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572602/real 1788572602] req@ffff94d408d3c700 x1875450488546048/t0(0) o400->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572618 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572618.949520:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3750 00000100:00080000:2.0:1788572618.949547:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788572618.949549:0:2403:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff94d408d3c700 x1875450488546048/t0(0) o400->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572618 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572618.949558:0:2403:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff94d405c94000 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:0.0:1788572618.949560:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:86f37580-247f-4c91-a842-1b0c4422007b:2404:1875450488545920:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572618.949571:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:86f37580-247f-4c91-a842-1b0c4422007b:2404:1875450488546176:192.168.202.125@tcp:400:kworker.0 00000100:00000400:1.0:1788572618.949571:0:2403:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572602/real 1788572602] req@ffff94d408d3c380 x1875450488546304/t0(0) o400->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572618 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572618.949576:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2405:1875450488547712:192.168.202.125@tcp:400:kworker.0 00000100:00100000:2.0:1788572618.949582:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488548352/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572618.949602:0:2403:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff94d408d3c380 x1875450488546304/t0(0) o400->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572618 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572618.949609:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572618.949611:0:2403:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from FULL to DISCONN 00000100:00080000:2.0:1788572618.949613:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572618.949618:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:02000400:1.0:1788572618.949621:0:2403:0:(import.c:182:ptlrpc_set_import_discon()) lustre-OST0001-osc-ffff94d402e39800: Connection to lustre-OST0001 (at 192.168.202.125@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1788572618.949623:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3750 00000100:00080000:1.0:1788572618.949628:0:2403:0:(import.c:418:ptlrpc_pinger_force()) lustre-OST0001_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1788572618.949633:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572618.949643:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572618.949648:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488548352:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788572618.949660:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488548480/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572618.949662:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2403:1875450488547072:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572618.949678:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2403:1875450488545792:192.168.202.125@tcp:400:kworker.0 00000100:00080000:2.0:1788572618.949679:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572618.949685:0:66171:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788572618.949687:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788572618.949691:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:86f37580-247f-4c91-a842-1b0c4422007b:2403:1875450488546048:192.168.202.125@tcp:400:kworker.0 00000100:00080000:2.0:1788572618.949692:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572618.949696:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788572618.949700:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:86f37580-247f-4c91-a842-1b0c4422007b:2403:1875450488546304:192.168.202.125@tcp:400:kworker.0 00000100:00080000:2.0:1788572618.949705:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3750 00000100:00080000:2.0:1788572618.949719:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572618.949725:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572618.949738:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488548608/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572618.949748:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488548480:192.168.202.125@tcp:38:kworker.0 00000100:00080000:2.0:1788572618.949749:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572618.949758:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572618.949759:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572618.949764:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3750 00000100:00080000:2.0:1788572618.949771:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572618.949777:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572618.949788:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488548736/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572618.949792:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488548608:192.168.202.125@tcp:8:kworker.0 00000100:00080000:2.0:1788572618.949810:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572618.949813:0:66171:0:(pinger.c:247:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788572618.949815:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572618.949818:0:66171:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1788572618.949843:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488548736:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572619.461295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572619.909353:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572619.973316:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572619.973326:0:66171:0:(pinger.c:247:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788572619.973330:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572619.973334:0:66171:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788572619.973337:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572619.973340:0:66171:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788572619.973343:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572619.973347:0:66171:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788572619.973349:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572619.973353:0:66171:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1788572620.485277:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572620.933361:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572621.509335:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572621.957400:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572622.533348:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572622.981392:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:1.0:1788572623.045353:0:2403:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572607/real 1788572607] req@ffff94d408d3dc00 x1875450488546560/t0(0) o400->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572623 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:0.0:1788572623.045392:0:2404:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572607/real 1788572607] req@ffff94d408d3e680 x1875450488546688/t0(0) o400->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572623 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572623.045406:0:2403:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff94d408d3dc00 x1875450488546560/t0(0) o400->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572623 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572623.045428:0:2404:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff94d408d3e680 x1875450488546688/t0(0) o400->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572623 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572623.045433:0:2403:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff94d411203000 already not connected (conn 2, was 1): CONNECTING 00000100:00080000:0.0:1788572623.045443:0:2404:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff94d405c94000 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:0.0:1788572623.045453:0:2404:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572607/real 1788572607] req@ffff94d408d3ea00 x1875450488546944/t0(0) o400->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572623 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:1.0:1788572623.045454:0:2403:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572607/real 1788572607] req@ffff94d408d3c000 x1875450488546816/t0(0) o400->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572623 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572623.045499:0:2404:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff94d408d3ea00 x1875450488546944/t0(0) o400->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572623 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572623.045507:0:2404:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff94d405c93800 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:1.0:1788572623.045514:0:2403:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff94d408d3c000 x1875450488546816/t0(0) o400->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572623 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572623.045524:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:86f37580-247f-4c91-a842-1b0c4422007b:2404:1875450488546688:192.168.202.125@tcp:400:kworker.0 00000100:00080000:1.0:1788572623.045526:0:2403:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff94d405c96800 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:0.0:1788572623.045536:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:86f37580-247f-4c91-a842-1b0c4422007b:2404:1875450488546944:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572623.045558:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:86f37580-247f-4c91-a842-1b0c4422007b:2403:1875450488546560:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572623.045569:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:86f37580-247f-4c91-a842-1b0c4422007b:2403:1875450488546816:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572623.557352:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00000100:00080000:0.0:1788572624.005384:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572624.005400:0:66171:0:(pinger.c:247:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572624.005403:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572624.005406:0:66171:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572624.005409:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572624.005412:0:66171:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572624.005416:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572624.005419:0:66171:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572624.005422:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572624.005424:0:66171:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1788572624.005444:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572624.581357:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572625.029411:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572625.093344:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572615/real 1788572615] req@ffff94d408c61880 x1875450488547968/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572625 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572625.093394:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488547968/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572625 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572625.093415:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572625.093437:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788572625.093443:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488547968:192.168.202.125@tcp:38:kworker.0 00000100:00080000:0.0:1788572625.093470:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572625.093477:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572625.093480:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572625.093483:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572625.093486:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572625.093489:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572625.093492:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572625.093497:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3768 00000100:00080000:0.0:1788572625.093508:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1788572625.093516:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572625.093528:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572625.093565:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488548864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572625.093586:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572625.093589:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572625.093591:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572625.093595:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1788572625.093619:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488548864:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572625.605330:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572626.053434:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572626.629328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572627.077352:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572627.653338:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572628.101392:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572628.165302:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572618/real 1788572618] req@ffff94d408c61c00 x1875450488548736/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572628 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572628.165342:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488548736/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572628 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:0.0:1788572628.165353:0:2403:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572612/real 1788572612] req@ffff94d408d3d500 x1875450488547328/t0(0) o400->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572628 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572628.165354:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572618/real 1788572618] req@ffff94d408c62a00 x1875450488548608/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572628 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572628.165392:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488548608/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572628 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572628.165401:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572618/real 1788572618] req@ffff94d408c63b80 x1875450488548480/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572628 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572628.165405:0:2403:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff94d408d3d500 x1875450488547328/t0(0) o400->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572628 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572628.165426:0:2403:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff94d405c94000 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:1.0:1788572628.165447:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488548480/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572628 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572628.165448:0:2403:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:86f37580-247f-4c91-a842-1b0c4422007b:2403:1875450488547328:192.168.202.125@tcp:400:kworker.0 00000100:00100000:1.0:1788572628.165469:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572618/real 1788572618] req@ffff94d408c61500 x1875450488548352/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572628 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:0.0:1788572628.165495:0:2404:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572612/real 1788572612] req@ffff94d408d3ce00 x1875450488547584/t0(0) o400->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572628 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572628.165496:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488548352/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572628 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572628.165527:0:2404:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff94d408d3ce00 x1875450488547584/t0(0) o400->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572628 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572628.165527:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572628.165537:0:2404:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff94d405c93800 already not connected (conn 2, was 1): DISCONN 00000100:00000400:0.0:1788572628.165554:0:2404:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572612/real 1788572612] req@ffff94d408d3df80 x1875450488547200/t0(0) o400->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572628 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572628.165563:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572628.165567:0:2404:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff94d408d3df80 x1875450488547200/t0(0) o400->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572628 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572628.165571:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488548736:192.168.202.125@tcp:8:kworker.0 00000100:00080000:1.0:1788572628.165584:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572628.165585:0:2404:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff94d411203000 already not connected (conn 2, was 1): CONNECTING 00000100:00080000:3.0:1788572628.165589:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000400:0.0:1788572628.165597:0:2404:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572612/real 1788572612] req@ffff94d408d3d180 x1875450488547456/t0(0) o400->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572628 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572628.165598:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00080000:3.0:1788572628.165598:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572628.165605:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572628.165614:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1788572628.165616:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488548608:192.168.202.125@tcp:8:kworker.0 00000100:00080000:3.0:1788572628.165620:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572628.165631:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788572628.165632:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572628.165644:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572628.165647:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00080000:3.0:1788572628.165651:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1788572628.165657:0:2404:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff94d408d3d180 x1875450488547456/t0(0) o400->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572628 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572628.165657:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488548480:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788572628.165657:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572628.165670:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3772 00000100:00080000:1.0:1788572628.165673:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572628.165685:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1788572628.165694:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00080000:0.0:1788572628.165699:0:2404:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff94d405c96800 already not connected (conn 3, was 1): CONNECTING 00000100:00080000:1.0:1788572628.165702:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572628.165706:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488548352:192.168.202.125@tcp:250:kworker.0 00000008:00100000:3.0:1788572628.165708:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572628.165715:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:86f37580-247f-4c91-a842-1b0c4422007b:2404:1875450488547584:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572628.165727:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:86f37580-247f-4c91-a842-1b0c4422007b:2404:1875450488547200:192.168.202.125@tcp:400:kworker.0 00000100:00100000:0.0:1788572628.165734:0:2404:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:86f37580-247f-4c91-a842-1b0c4422007b:2404:1875450488547456:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572628.165743:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488548992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572628.165770:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572628.165774:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572628.165776:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572628.165781:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3772 00000100:00080000:3.0:1788572628.165787:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1788572628.165791:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572628.165797:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572628.165808:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488548992:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572628.165813:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488549120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572628.165833:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572628.165836:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572628.165839:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572628.165843:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3772 00000100:00080000:3.0:1788572628.165850:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1788572628.165854:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788572628.165869:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488549248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572628.165880:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572628.165883:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572628.165885:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572628.165889:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3772 00000100:00080000:3.0:1788572628.165894:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1788572628.165898:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572628.165905:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572628.165914:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488549120:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572628.165930:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488549376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572628.165948:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788572628.165950:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488549248:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788572628.165956:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1788572628.165999:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488549376:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572628.677277:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572629.125329:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572629.701313:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572630.149392:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572630.213340:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572630.213352:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572630.213357:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572630.213362:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572630.213366:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572630.213370:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572630.213373:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572630.213376:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572630.213379:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572630.213382:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1788572630.725331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572631.173365:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1788572631.749334:0:2405:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572615/real 1788572615] req@ffff94d408c61f80 x1875450488548096/t0(0) o400->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572631 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:3.0:1788572631.749334:0:2406:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572615/real 1788572615] req@ffff94d408c63800 x1875450488547840/t0(0) o400->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572631 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572631.749354:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00000100:00100000:2.0:1788572631.749387:0:2405:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff94d408c61f80 x1875450488548096/t0(0) o400->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572631 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572631.749394:0:2406:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff94d408c63800 x1875450488547840/t0(0) o400->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572631 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572631.749397:0:2405:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff94d405c96800 already not connected (conn 3, was 1): CONNECTING 00000100:00080000:3.0:1788572631.749414:0:2406:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff94d411203000 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:2.0:1788572631.749419:0:2405:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:86f37580-247f-4c91-a842-1b0c4422007b:2405:1875450488548096:192.168.202.125@tcp:400:kworker.0 00000100:00000400:3.0:1788572631.749454:0:2406:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572615/real 1788572615] req@ffff94d408c60e00 x1875450488548224/t0(0) o400->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572631 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572631.749470:0:2406:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff94d408c60e00 x1875450488548224/t0(0) o400->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572631 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572631.749477:0:2406:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff94d405c93800 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:3.0:1788572631.749486:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:86f37580-247f-4c91-a842-1b0c4422007b:2406:1875450488547840:192.168.202.125@tcp:400:kworker.0 00000100:00100000:3.0:1788572631.749493:0:2406:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:86f37580-247f-4c91-a842-1b0c4422007b:2406:1875450488548224:192.168.202.125@tcp:400:kworker.0 00010000:00010000:1.0:1788572632.197449:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572632.773365:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00000100:00080000:3.0:1788572633.221303:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572633.221312:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572633.221315:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572633.221318:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572633.221320:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572633.221323:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572633.221325:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572633.221327:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572633.221329:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572633.221330:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1788572633.221425:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572633.797335:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572634.245402:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572634.821271:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572635.269518:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572635.845341:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572636.293429:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572636.869269:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572637.317467:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572637.893320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572638.341411:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572638.405317:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572638.405325:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572638.405328:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572638.405331:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572638.405334:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572638.405336:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572638.405338:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572638.405341:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572638.405344:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572638.405346:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1788572638.917347:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572639.365408:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572639.941300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572640.389347:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572640.453303:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572625/real 1788572625] req@ffff94d408d3ea00 x1875450488548864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572640 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572640.453345:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488548864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572640 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572640.453374:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572640.453392:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572640.453396:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488548864:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788572640.453422:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572640.453428:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572640.453431:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572640.453433:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572640.453435:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572640.453438:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572640.453440:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572640.453446:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3778 00000100:00080000:3.0:1788572640.453456:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1788572640.453463:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572640.453472:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572640.453499:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488549504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572640.453518:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572640.453521:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572640.453523:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572640.453525:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1788572640.453548:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488549504:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572640.965303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572641.413360:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572641.989333:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572642.437351:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572643.013325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572643.461445:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572643.525342:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572628/real 1788572628] req@ffff94d408c57100 x1875450488549376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572643 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572643.525379:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488549376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572643 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572643.525394:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572628/real 1788572628] req@ffff94d408c55880 x1875450488549120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572643 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572643.525408:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488549120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572643 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572643.525420:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572628/real 1788572628] req@ffff94d408c56a00 x1875450488549248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572643 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572643.525442:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488549248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572643 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572643.525458:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572628/real 1788572628] req@ffff94d408c55180 x1875450488548992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572643 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572643.525473:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488548992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572643 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572643.525497:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572643.525525:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572643.525530:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488549376:192.168.202.125@tcp:38:kworker.0 00000100:00080000:1.0:1788572643.525536:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572643.525545:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572643.525553:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488549120:192.168.202.125@tcp:8:kworker.0 00000100:00080000:1.0:1788572643.525559:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572643.525568:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572643.525572:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488549248:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788572643.525576:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572643.525593:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572643.525598:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572643.525607:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572643.525620:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00080000:0.0:1788572643.525621:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3781 00000100:00100000:1.0:1788572643.525628:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488548992:192.168.202.125@tcp:8:kworker.0 00000100:00080000:0.0:1788572643.525642:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 10s 00000100:00080000:0.0:1788572643.525657:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788572643.525697:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488549632/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572643.525725:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572643.525728:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572643.525730:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572643.525733:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3781 00000100:00080000:0.0:1788572643.525738:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 10s 00000100:00080000:0.0:1788572643.525745:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572643.525755:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572643.525768:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488549632:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788572643.525769:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488549760/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572643.525795:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572643.525797:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572643.525799:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572643.525802:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572643.525804:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572643.525808:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3781 00000100:00080000:0.0:1788572643.525810:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 10s 00000100:00080000:0.0:1788572643.525814:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572643.525817:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572643.525828:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488549888/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572643.525835:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572643.525836:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572643.525838:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572643.525840:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3781 00000100:00080000:0.0:1788572643.525855:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 10s 00000100:00080000:0.0:1788572643.525861:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572643.525867:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572643.525881:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488550016/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572643.525900:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572643.525905:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1788572643.525940:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488549760:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572643.525982:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488549888:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572643.526014:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488550016:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572644.037274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572644.485357:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572645.061274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572645.509369:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572645.573275:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572645.573287:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572645.573290:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572645.573294:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572645.573298:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572645.573301:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572645.573304:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572645.573309:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572645.573313:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572645.573317:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1788572646.085268:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572646.533325:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572647.109309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572647.557377:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572647.621380:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572647.621388:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572647.621391:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572647.621393:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572647.621395:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572647.621397:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572647.621399:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572647.621401:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572647.621402:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572647.621404:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1788572648.133287:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572648.581504:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572649.157317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572649.605374:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572650.181290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572650.629380:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572651.205311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572651.653375:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572652.229312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572652.677399:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572652.741394:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572652.741405:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572652.741408:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572652.741423:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572652.741426:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572652.741429:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572652.741432:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572652.741434:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572652.741437:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572652.741440:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1788572653.253377:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572653.701463:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572654.277377:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572654.725413:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572655.301341:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572655.749374:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572656.325326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572656.773361:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572657.349310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572657.797368:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572657.861259:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572657.861267:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572657.861271:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572657.861275:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572657.861278:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572657.861280:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572657.861282:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572657.861284:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572657.861288:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572657.861291:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1788572658.373323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572658.821369:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572659.397271:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572659.845352:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572660.421300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572660.869330:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572660.933297:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572640/real 1788572640] req@ffff94d408c56300 x1875450488549504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572660 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572660.933328:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488549504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572660 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572660.933344:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572660.933358:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572660.933361:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488549504:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788572660.933378:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572660.933384:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572660.933387:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572660.933391:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572660.933394:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572660.933397:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572660.933400:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572660.933406:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3793 00000100:00080000:3.0:1788572660.933416:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1788572660.933423:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572660.933433:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572660.933461:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488550144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572660.933482:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572660.933486:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572660.933489:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572660.933491:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1788572660.933509:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488550144:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572661.445343:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00000800:00000100:3.0:1788572661.573319:0:2393:0:(socklnd_cb.c:2247:ksocknal_find_timed_out_conn()) A connection with 12345-192.168.202.125@tcp (192.168.202.125:988) timed out; the network or node may be down. 00010000:00010000:0.0:1788572661.893433:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572662.469290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572662.917335:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572663.493295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572663.941346:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572664.005295:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572643/real 1788572643] req@ffff94d408d3d180 x1875450488549760/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572663 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572664.005327:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488549760/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572663 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572664.005338:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572643/real 1788572643] req@ffff94d408d3df80 x1875450488549888/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572663 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572664.005349:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488549888/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572663 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572664.005359:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572643/real 1788572643] req@ffff94d408d3ce00 x1875450488550016/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572663 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572664.005376:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488550016/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572663 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572664.005382:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572643/real 1788572643] req@ffff94d408d3ea00 x1875450488549632/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572663 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572664.005389:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488549632/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572663 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572664.005402:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572664.005417:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572664.005423:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488549760:192.168.202.125@tcp:38:kworker.0 00000100:00080000:1.0:1788572664.005430:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572664.005439:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572664.005442:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488549888:192.168.202.125@tcp:8:kworker.0 00000100:00080000:3.0:1788572664.005445:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572664.005457:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572664.005458:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572664.005464:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572664.005467:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572664.005469:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572664.005472:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00080000:3.0:1788572664.005475:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3796 00000100:00100000:1.0:1788572664.005479:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488550016:192.168.202.125@tcp:8:kworker.0 00000100:00080000:1.0:1788572664.005484:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572664.005485:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 15s 00000100:00080000:1.0:1788572664.005489:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572664.005493:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488549632:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788572664.005495:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572664.005506:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572664.005531:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488550272/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572664.005553:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572664.005555:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572664.005557:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572664.005560:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572664.005563:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572664.005567:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3796 00000100:00080000:3.0:1788572664.005572:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1788572664.005576:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572664.005581:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572664.005586:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488550272:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788572664.005595:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488550400/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572664.005606:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572664.005610:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572664.005611:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572664.005615:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3796 00000100:00080000:3.0:1788572664.005619:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1788572664.005623:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572664.005628:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572664.005640:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488550528/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572664.005653:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572664.005655:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572664.005656:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572664.005658:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3796 00000100:00080000:3.0:1788572664.005662:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1788572664.005666:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788572664.005673:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488550400:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572664.005678:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488550656/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572664.005686:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572664.005689:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1788572664.005699:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488550528:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572664.005714:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488550656:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788572664.517339:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572664.965379:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572665.541306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572665.989369:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572666.053315:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572666.053327:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572666.053330:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572666.053333:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572666.053335:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572666.053339:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572666.053341:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572666.053344:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572666.053345:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572666.053347:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1788572666.565322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00000400:00000100:2.0:1788572666.693562:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.202.125:988 00000400:00000100:2.0:1788572666.693610:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788572666.693628:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788572666.693668:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788572666.693674:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788572666.693681:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:1.0:1788572666.694008:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572664/real 1788572666] req@ffff94d408c57480 x1875450488550656/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572666.694050:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488550656/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572666.694069:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572666.694077:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572666.694082:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488550656:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788572666.694093:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572664/real 1788572666] req@ffff94d408c54000 x1875450488550400/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572666.694110:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488550400/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572666.694121:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572666.694140:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572666.694145:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488550400:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572666.694155:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572664/real 1788572666] req@ffff94d408c54700 x1875450488550528/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572666.694169:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488550528/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572666.694179:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572666.694183:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572666.694186:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488550528:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572666.694241:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572664/real 1788572666] req@ffff94d408c55f80 x1875450488550272/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572666.694262:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488550272/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572666.694272:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572666.694278:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572666.694282:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488550272:192.168.202.125@tcp:38:kworker.0 00010000:00010000:1.0:1788572667.013389:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572667.589305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572668.037370:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572668.613361:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00000100:00080000:0.0:1788572669.061369:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572669.061378:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572669.061382:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572669.061391:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3817 00000100:00080000:0.0:1788572669.061401:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572669.061408:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788572669.061444:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488550784/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572669.061472:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572669.061475:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572669.061477:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1788572669.061485:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572669.061500:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3817 00000100:00080000:0.0:1788572669.061504:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572669.061507:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572669.061517:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572669.061524:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488550784:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788572669.061529:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488550912/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572669.061545:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572669.061548:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572669.061550:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572669.061552:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572669.061553:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572669.061556:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3817 00000100:00080000:0.0:1788572669.061564:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572669.061569:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572669.061572:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572669.061585:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f100 x1875450488551040/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572669.061595:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572669.061599:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572669.061600:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572669.061601:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3817 00000100:00080000:0.0:1788572669.061604:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572669.061606:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572669.061610:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572669.061616:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488551168/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572669.061679:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488550912:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788572669.061709:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488551040:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788572669.061734:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488551168:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572669.637308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00000400:00000100:1.0:1788572669.765617:0:2392:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.202.125:988 00000400:00000100:1.0:1788572669.765679:0:2392:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788572669.765701:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788572669.765752:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788572669.765769:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788572669.765782:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:2.0:1788572669.765793:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572669/real 1788572669] req@ffff94d408d3e680 x1875450488550912/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572669.765830:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488550912/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572669.765844:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572669.765851:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788572669.765856:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488550912:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788572669.765866:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572669/real 1788572669] req@ffff94d408d3f100 x1875450488551040/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572669.765899:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f100 x1875450488551040/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572669.765909:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572669.765914:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788572669.765918:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488551040:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788572669.765926:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572669/real 1788572669] req@ffff94d408d3ca80 x1875450488551168/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572669.765941:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488551168/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572669.765950:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572669.765956:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788572669.765959:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488551168:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788572669.765968:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572669/real 1788572669] req@ffff94d408d3d500 x1875450488550784/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572669.765982:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488550784/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572669.765992:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572669.765996:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788572669.766000:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488550784:192.168.202.125@tcp:250:kworker.0 00010000:00010000:0.0:1788572670.085410:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572670.661344:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572671.109391:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572671.685707:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572672.133424:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572672.709555:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572673.157352:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572673.733353:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572674.181401:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572674.245318:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572674.245330:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572674.245334:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572674.245344:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3822 00000100:00080000:3.0:1788572674.245355:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572674.245362:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788572674.245399:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488551296/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572674.245431:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572674.245435:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572674.245438:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572674.245443:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3822 00000100:00080000:3.0:1788572674.245447:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572674.245450:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572674.245458:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572674.245469:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488551424/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572674.245478:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572674.245482:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1788572674.245484:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488551296:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788572674.245485:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572674.245493:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572674.245495:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572674.245499:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3822 00000100:00080000:3.0:1788572674.245501:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572674.245503:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572674.245507:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572674.245515:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488551552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572674.245522:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572674.245525:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572674.245526:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572674.245528:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3822 00000100:00080000:3.0:1788572674.245531:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572674.245535:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572674.245538:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572674.245545:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488551680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572674.245574:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488551424:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572674.245596:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488551552:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572674.245610:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488551680:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572674.757330:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572675.205335:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572675.781277:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00000400:00000100:2.0:1788572675.909479:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.202.125:988 00000400:00000100:2.0:1788572675.909515:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788572675.909528:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788572675.909558:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788572675.909564:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788572675.909569:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:1.0:1788572675.909620:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572674/real 1788572675] req@ffff94d408c54700 x1875450488551424/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572675.909651:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488551424/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572675.909663:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572675.909669:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572675.909673:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488551424:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572675.909681:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572674/real 1788572675] req@ffff94d408c54000 x1875450488551552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572675.909693:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488551552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572675.909701:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572675.909705:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572675.909709:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488551552:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572675.909716:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572674/real 1788572675] req@ffff94d408c57480 x1875450488551680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572675.909728:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488551680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572675.909741:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572675.909745:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572675.909749:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488551680:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572675.909757:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572674/real 1788572675] req@ffff94d408c55f80 x1875450488551296/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572675.909770:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488551296/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572675.909777:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572675.909780:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572675.909784:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488551296:192.168.202.125@tcp:250:kworker.0 00010000:00010000:0.0:1788572676.229368:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572676.805282:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572677.253337:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572677.829287:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572678.277363:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572678.853818:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572679.301319:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572679.365271:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572679.365280:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572679.365284:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572679.365294:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3827 00000100:00080000:3.0:1788572679.365311:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572679.365321:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788572679.365353:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488551808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572679.365373:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572679.365376:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572679.365378:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572679.365381:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3827 00000100:00080000:3.0:1788572679.365383:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572679.365385:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572679.365392:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572679.365403:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488551936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572679.365413:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488551808:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788572679.365419:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572679.365423:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572679.365426:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572679.365428:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572679.365429:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572679.365433:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3827 00000100:00080000:3.0:1788572679.365437:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572679.365439:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572679.365443:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572679.365452:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488552064/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572679.365460:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572679.365467:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572679.365469:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788572679.365473:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788572679.365476:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3827 00000100:00080000:3.0:1788572679.365480:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572679.365484:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572679.365488:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572679.365499:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488552192/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572679.365505:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572679/real 1788572679] req@ffff94d408c56300 x1875450488551808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572679.365520:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488551808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572679.365527:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572679.365530:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572679.365533:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488551808:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788572679.365548:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488551936:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572679.365565:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572679.365573:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572679/real 1788572679] req@ffff94d408c55180 x1875450488551936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572679.365582:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488551936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572679.365587:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572679.365590:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572679.365592:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488551936:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572679.365596:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488552064:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572679.365607:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572679.365619:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572679/real 1788572679] req@ffff94d408c56a00 x1875450488552064/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572679.365627:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488552064/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572679.365632:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572679.365634:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572679.365637:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488552064:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572679.365640:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488552192:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572679.365650:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572679.365656:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572679/real 1788572679] req@ffff94d408c55880 x1875450488552192/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572679.365668:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488552192/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572679.365673:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572679.365674:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572679.365676:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488552192:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572679.877303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572680.325435:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572680.901354:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572681.349348:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572681.925788:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572682.373367:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572682.949293:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572683.397379:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572683.973400:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572684.421424:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572684.485287:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572684.485309:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572684.485314:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572684.485326:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3832 00000100:00080000:3.0:1788572684.485340:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572684.485349:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788572684.485395:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488552320/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572684.485425:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572684.485428:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572684.485429:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572684.485432:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3832 00000100:00080000:3.0:1788572684.485435:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572684.485437:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572684.485447:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572684.485460:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488552448/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572684.485468:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788572684.485474:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488552320:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788572684.485475:0:36550:0:(pinger.c:247:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572684.485480:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572684.485483:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572684.485485:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572684.485488:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3832 00000100:00080000:3.0:1788572684.485490:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572684.485493:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572684.485497:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572684.485506:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488552576/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572684.485513:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572684.485516:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572684.485519:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572684.485523:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3832 00000100:00080000:3.0:1788572684.485528:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572684.485533:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572684.485539:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572684.485553:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488552704/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572684.485590:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488552448:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572684.485617:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488552576:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572684.485638:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488552704:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572684.997340:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00000400:00000100:0.0:1788572685.125688:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.202.125:988 00000400:00000100:0.0:1788572685.125773:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788572685.125798:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:0.0:1788572685.125904:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:0.0:1788572685.125918:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:0.0:1788572685.125931:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:1.0:1788572685.126136:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572684/real 1788572685] req@ffff94d408c54380 x1875450488552448/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572685.126244:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488552448/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572685.126264:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572685.126271:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572685.126277:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488552448:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572685.126287:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572684/real 1788572685] req@ffff94d408c57800 x1875450488552576/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572685.126303:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488552576/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572685.126314:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572685.126318:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572685.126322:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488552576:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572685.126329:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572684/real 1788572685] req@ffff94d408c56d80 x1875450488552704/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572685.126344:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488552704/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572685.126353:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572685.126357:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572685.126362:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488552704:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572685.126371:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572684/real 1788572685] req@ffff94d408c57100 x1875450488552320/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572685.126383:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488552320/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572685.126391:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572685.126394:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572685.126397:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488552320:192.168.202.125@tcp:250:kworker.0 00010000:00010000:3.0:1788572685.445384:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572685.493313:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572660/real 1788572660] req@ffff94d408c55c00 x1875450488550144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572685 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572685.493343:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488550144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572685 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572685.493359:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572685.493378:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572685.493381:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488550144:192.168.202.125@tcp:38:kworker.0 00000100:00080000:0.0:1788572685.493427:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572685.493436:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572685.493440:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572685.493449:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3837 00000100:00080000:0.0:1788572685.493460:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572685.493469:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788572685.493510:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f800 x1875450488552832/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572685.493537:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572685.493541:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572685.493542:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572685.493545:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3837 00000100:00080000:0.0:1788572685.493548:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572685.493550:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572685.493562:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572685.493574:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488552960/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572685.493583:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488552832:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788572685.493598:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572685.493603:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572685.493606:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572685.493612:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3814 00000100:00080000:0.0:1788572685.493619:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572685.493625:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572685.493633:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572685.493650:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488553088/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788572685.493655:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788572685.493671:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572685.493677:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572685.493679:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572685.493686:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3837 00000100:00080000:0.0:1788572685.493697:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788572685.493698:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572685/real 1788572685] req@ffff94d408d3f800 x1875450488552832/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572685.493711:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788572685.493718:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f800 x1875450488552832/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788572685.493721:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788572685.493740:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572685.493745:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572685.493749:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f100 x1875450488553216/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572685.493754:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488552832:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788572685.493782:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572685.493785:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572685.493787:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788572685.493791:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488552960:192.168.202.125@tcp:38:kworker.0 00000100:00080000:0.0:1788572685.493797:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3837 00000100:00080000:0.0:1788572685.493821:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572685.493828:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572685.493835:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788572685.493841:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572685.493857:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572685/real 1788572685] req@ffff94d408d3d500 x1875450488552960/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572685.493865:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488553344/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572685.493881:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488552960/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572685.493896:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572685.493901:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572685.493907:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488552960:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572685.493917:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488553088:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572685.493941:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572685.493951:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572685/real 1788572685] req@ffff94d408d3ca80 x1875450488553088/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572685.493959:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488553088/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572685.493965:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572685.493967:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572685.493969:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488553088:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572685.493974:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488553216:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572685.493986:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572685.493992:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572685/real 1788572685] req@ffff94d408d3f100 x1875450488553216/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572685.493999:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f100 x1875450488553216/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572685.494005:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572685.494007:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572685.494009:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488553216:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572685.494020:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488553344:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572685.494036:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572685.494042:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572685/real 1788572685] req@ffff94d408d3e680 x1875450488553344/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572685.494050:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488553344/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572685.494055:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572685.494057:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572685.494059:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488553344:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572686.021823:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572686.469411:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572687.045481:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572687.493426:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572688.069302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572688.517447:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572689.093406:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572689.541402:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572690.117336:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572690.565434:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572690.629315:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572690.629327:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572690.629332:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572690.629344:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3838 00000100:00080000:3.0:1788572690.629357:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572690.629367:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788572690.629413:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488553472/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572690.629448:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572690.629452:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572690.629454:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572690.629459:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3838 00000100:00080000:3.0:1788572690.629463:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572690.629468:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572690.629481:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572690.629498:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488553600/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572690.629508:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488553472:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788572690.629518:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572690.629522:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572690.629525:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572690.629529:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3838 00000100:00080000:3.0:1788572690.629533:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572690.629537:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572690.629542:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572690.629553:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55500 x1875450488553728/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572690.629562:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572690.629565:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572690.629567:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572690.629570:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3838 00000100:00080000:3.0:1788572690.629573:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572690.629578:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572690.629583:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572690.629592:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488553856/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788572690.629595:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788572690.629601:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572690.629604:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572690.629606:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572690.629610:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3838 00000100:00080000:3.0:1788572690.629613:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572690.629616:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572690.629620:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572690.629630:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488553984/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572690.629640:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572690/real 1788572690] req@ffff94d408c57b80 x1875450488553472/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572690.629662:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488553472/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572690.629685:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572690.629689:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572690.629693:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488553472:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788572690.629720:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488553600:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572690.629751:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572690.629766:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572690/real 1788572690] req@ffff94d408c56680 x1875450488553600/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572690.629780:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488553600/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572690.629802:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572690.629806:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572690.629810:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488553600:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572690.629815:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488553728:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572690.629833:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572690.629842:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572690/real 1788572690] req@ffff94d408c55500 x1875450488553728/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572690.629850:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55500 x1875450488553728/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572690.629856:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572690.629858:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572690.629860:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488553728:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572690.629864:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488553856:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572690.629876:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572690.629882:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572690/real 1788572690] req@ffff94d408c54a80 x1875450488553856/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572690.629894:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488553856/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572690.629902:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572690.629906:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572690.629910:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488553856:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572690.629917:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488553984:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572690.629937:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572690.629946:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572690/real 1788572690] req@ffff94d408c54e00 x1875450488553984/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572690.629954:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488553984/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572690.629960:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572690.629962:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572690.629964:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488553984:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572691.141354:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572691.589382:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572692.166118:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572692.613347:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572693.189628:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572693.637374:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572694.214102:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572694.661377:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572695.237318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572695.685443:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572695.749296:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572695.749313:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572695.749319:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572695.749334:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3844 00000100:00080000:3.0:1788572695.749350:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572695.749359:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788572695.749413:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488554112/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572695.749444:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572695.749447:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572695.749448:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572695.749452:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3844 00000100:00080000:3.0:1788572695.749455:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572695.749458:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572695.749473:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572695.749484:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488554240/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572695.749492:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488554112:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788572695.749496:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572695.749500:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572695.749501:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572695.749505:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3844 00000100:00080000:3.0:1788572695.749506:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572695.749509:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572695.749512:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572695.749522:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55500 x1875450488554368/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572695.749529:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572695.749531:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572695.749532:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572695.749535:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3844 00000100:00080000:3.0:1788572695.749538:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572695.749541:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572695.749545:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572695.749566:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488554496/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572695.749574:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572695.749577:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572695.749578:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572695.749581:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3844 00000100:00080000:3.0:1788572695.749584:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572695.749588:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572695.749591:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572695.749609:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488554624/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572695.749624:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488554240:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572695.749652:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488554368:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572695.749672:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488554496:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572695.749694:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488554624:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788572696.261292:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572696.709356:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572697.285291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00000400:00000100:2.0:1788572697.413468:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.202.125:988 00000400:00000100:2.0:1788572697.413504:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788572697.413532:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788572697.413564:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788572697.413571:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788572697.413576:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788572697.413581:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:1.0:1788572697.413666:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572695/real 1788572697] req@ffff94d408c54a80 x1875450488554240/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572697.413710:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488554240/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572697.413731:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572697.413738:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572697.413744:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488554240:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572697.413755:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572695/real 1788572697] req@ffff94d408c55500 x1875450488554368/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572697.413780:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55500 x1875450488554368/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572697.413791:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572697.413795:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572697.413798:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488554368:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572697.413808:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572695/real 1788572697] req@ffff94d408c56680 x1875450488554496/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572697.413823:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488554496/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572697.413832:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572697.413837:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572697.413840:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488554496:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572697.413849:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572695/real 1788572697] req@ffff94d408c57b80 x1875450488554624/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572697.413863:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488554624/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572697.413874:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572697.413878:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572697.413882:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488554624:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572697.413890:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572695/real 1788572697] req@ffff94d408c54e00 x1875450488554112/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572697.413904:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488554112/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572697.413913:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572697.413917:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572697.413920:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488554112:192.168.202.125@tcp:250:kworker.0 00010000:00010000:3.0:1788572697.733357:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572698.309282:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572698.757388:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572699.333286:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572699.781358:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572700.357303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572700.805369:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572700.869258:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572700.869265:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572700.869269:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572700.869278:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3849 00000100:00080000:3.0:1788572700.869288:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572700.869295:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788572700.869329:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488554752/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572700.869356:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572700.869359:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572700.869361:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572700.869363:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3849 00000100:00080000:3.0:1788572700.869366:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572700.869369:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572700.869376:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572700.869386:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488554880/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572700.869395:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488554752:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788572700.869396:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572700.869405:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572700.869407:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572700.869410:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3849 00000100:00080000:3.0:1788572700.869412:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572700.869415:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572700.869430:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572700.869441:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488555008/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572700.869448:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572700.869450:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572700.869452:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572700.869454:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3849 00000100:00080000:3.0:1788572700.869459:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572700.869463:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572700.869473:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572700.869484:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488555136/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788572700.869485:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788572700.869494:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572700.869497:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572700.869498:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572700.869501:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3849 00000100:00080000:3.0:1788572700.869505:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572700.869508:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572700.869512:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572700.869523:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488555264/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572700.869524:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572700/real 1788572700] req@ffff94d408c55c00 x1875450488554752/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572700.869559:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488554752/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572700.869577:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572700.869583:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572700.869587:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488554752:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788572700.869615:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488554880:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572700.869643:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572700.869658:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572700/real 1788572700] req@ffff94d408c57100 x1875450488554880/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572700.869670:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488554880/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572700.869678:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572700.869681:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572700.869684:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488554880:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572700.869691:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488555008:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572700.869710:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572700.869721:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572700/real 1788572700] req@ffff94d408c56d80 x1875450488555008/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572700.869734:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488555008/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572700.869742:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572700.869745:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572700.869748:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488555008:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572700.869755:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488555136:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572700.869775:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572700.869786:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572700/real 1788572700] req@ffff94d408c57800 x1875450488555136/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572700.869804:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488555136/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572700.869812:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572700.869815:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572700.869818:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488555136:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572700.869823:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488555264:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572700.869843:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572700.869853:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572700/real 1788572700] req@ffff94d408c54380 x1875450488555264/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572700.869864:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488555264/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572700.869884:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572700.869887:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572700.869890:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488555264:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788572701.381295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572701.829334:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572702.405345:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572702.853350:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572703.429284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572703.877370:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572704.453656:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572704.901399:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572705.477839:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572705.925461:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572705.989280:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572705.989301:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572705.989307:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572705.989320:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3854 00000100:00080000:3.0:1788572705.989333:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572705.989349:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788572705.989401:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488555392/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572705.989435:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572705.989440:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572705.989443:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572705.989447:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3854 00000100:00080000:3.0:1788572705.989452:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572705.989458:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572705.989470:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572705.989487:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488555392:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788572705.989491:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488555520/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572705.989506:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572705.989510:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572705.989513:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572705.989517:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3854 00000100:00080000:3.0:1788572705.989522:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572705.989527:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572705.989532:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572705.989543:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488555648/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572705.989551:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572705.989554:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572705.989556:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788572705.989563:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788572705.989564:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3854 00000100:00080000:3.0:1788572705.989572:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572705.989578:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572705.989585:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572705.989599:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488555776/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572705.989603:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572705/real 1788572705] req@ffff94d408c55880 x1875450488555392/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572705.989629:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788572705.989636:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488555392/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572705.989636:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572705.989642:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572705.989654:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3854 00000100:00080000:1.0:1788572705.989664:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572705.989664:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572705.989673:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:3.0:1788572705.989673:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788572705.989680:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488555392:192.168.202.125@tcp:250:kworker.0 00000008:00100000:3.0:1788572705.989684:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572705.989701:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488555904/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572705.989706:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488555520:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572705.989734:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572705.989744:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572705/real 1788572705] req@ffff94d408c56a00 x1875450488555520/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572705.989753:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488555520/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572705.989761:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572705.989765:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572705.989767:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488555520:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572705.989772:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488555648:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572705.989784:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572705.989790:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572705/real 1788572705] req@ffff94d408c55180 x1875450488555648/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572705.989799:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488555648/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572705.989806:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572705.989809:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572705.989812:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488555648:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572705.989817:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488555776:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572705.989833:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572705.989840:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572705/real 1788572705] req@ffff94d408c56300 x1875450488555776/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572705.989851:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488555776/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572705.989858:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572705.989861:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572705.989863:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488555776:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572705.989875:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488555904:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572705.989890:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572705.989896:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572705/real 1788572705] req@ffff94d408c55f80 x1875450488555904/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572705.989904:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488555904/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572705.989910:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572705.989913:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572705.989914:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488555904:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788572706.501556:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572706.949369:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572707.525548:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572707.973370:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572708.549355:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572708.997433:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572709.573449:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572710.021444:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572710.597270:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572711.045382:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572711.109294:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572711.109317:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572711.109321:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572711.109332:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3859 00000100:00080000:2.0:1788572711.109344:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572711.109352:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788572711.109391:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488556032/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572711.109421:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572711.109426:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572711.109428:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572711.109433:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3859 00000100:00080000:2.0:1788572711.109437:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572711.109442:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572711.109454:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572711.109470:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488556160/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572711.109481:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488556032:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788572711.109483:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572711.109492:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572711.109494:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572711.109498:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3859 00000100:00080000:2.0:1788572711.109501:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572711.109504:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572711.109509:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572711.109519:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488556288/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572711.109526:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572711.109529:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572711.109530:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572711.109533:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3859 00000100:00080000:2.0:1788572711.109535:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572711.109538:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572711.109541:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572711.109549:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63100 x1875450488556416/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572711.109555:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572711.109558:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572711.109559:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572711.109562:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3859 00000100:00080000:2.0:1788572711.109565:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572711.109567:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572711.109571:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788572711.109577:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788572711.109592:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60e00 x1875450488556544/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572711.109617:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572711/real 1788572711] req@ffff94d408c61f80 x1875450488556032/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572711.109636:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488556032/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572711.109645:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572711.109649:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572711.109652:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488556032:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788572711.109682:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488556160:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572711.109709:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572711.109723:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572711/real 1788572711] req@ffff94d408c61880 x1875450488556160/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572711.109733:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488556160/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572711.109739:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572711.109742:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572711.109753:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488556160:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572711.109759:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488556288:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572711.109775:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572711.109783:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572711/real 1788572711] req@ffff94d408c63480 x1875450488556288/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572711.109804:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488556288/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572711.109819:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572711.109823:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572711.109827:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488556288:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572711.109834:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488556416:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572711.109862:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572711.109874:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572711/real 1788572711] req@ffff94d408c63100 x1875450488556416/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572711.109888:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63100 x1875450488556416/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572711.109897:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572711.109901:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572711.109905:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488556416:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572711.109912:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488556544:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572711.109935:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572711.109945:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572711/real 1788572711] req@ffff94d408c60e00 x1875450488556544/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572711.109954:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60e00 x1875450488556544/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572711.109959:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572711.109961:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572711.109964:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488556544:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572711.621300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572712.069355:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572712.645270:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572713.093401:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572713.669279:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572714.117334:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572714.693272:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572715.141389:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572715.717283:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572716.165326:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572716.229300:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572716.229316:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572716.229321:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572716.229334:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3864 00000100:00080000:2.0:1788572716.229349:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572716.229359:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788572716.229412:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488556672/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572716.229448:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572716.229453:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572716.229455:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572716.229459:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3864 00000100:00080000:2.0:1788572716.229464:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572716.229469:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572716.229480:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572716.229496:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488556800/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572716.229503:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488556672:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788572716.229515:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572716.229520:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572716.229523:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572716.229528:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3864 00000100:00080000:2.0:1788572716.229534:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572716.229538:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572716.229545:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572716.229562:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488556928/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572716.229573:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572716.229577:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572716.229579:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572716.229584:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3864 00000100:00080000:2.0:1788572716.229589:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572716.229593:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572716.229600:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572716.229613:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488557056/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572716.229621:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572716.229625:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572716.229626:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572716.229629:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3864 00000100:00080000:2.0:1788572716.229632:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572716.229639:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788572716.229640:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488556800:192.168.202.125@tcp:38:kworker.0 00000008:00100000:2.0:1788572716.229644:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572716.229656:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488557184/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572716.229691:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488556928:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572716.229717:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488557056:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572716.229748:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488557184:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572716.741315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572717.189388:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572717.765256:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572718.213349:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572718.789328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00000400:00000100:0.0:1788572718.917595:0:2392:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:0.0:1788572718.917647:0:2392:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788572718.917666:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:0.0:1788572718.917716:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:0.0:1788572718.917726:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:0.0:1788572718.917735:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:0.0:1788572718.917743:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:1.0:1788572718.917824:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572716/real 1788572718] req@ffff94d408c61c00 x1875450488557184/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572718.917852:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488557184/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572718.917863:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572718.917867:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572718.917871:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488557184:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572718.917879:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572716/real 1788572718] req@ffff94d408c61500 x1875450488556800/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572718.917891:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488556800/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572718.917899:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572718.917902:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572718.917905:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488556800:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572718.917912:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572716/real 1788572718] req@ffff94d408c63b80 x1875450488556928/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572718.917923:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488556928/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572718.917931:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572718.917935:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572718.917939:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488556928:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572718.917946:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572716/real 1788572718] req@ffff94d408c62a00 x1875450488557056/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572718.917958:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488557056/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572718.917966:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572718.917984:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572718.917988:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488557056:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572718.917995:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572716/real 1788572718] req@ffff94d408c63800 x1875450488556672/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572718.918009:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488556672/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572718.918016:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572718.918020:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572718.918024:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488556672:192.168.202.125@tcp:250:kworker.0 00010000:00010000:0.0:1788572719.237415:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572719.813308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572720.261337:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572720.837289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572721.285405:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572721.349343:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572721.349356:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572721.349368:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572721.349378:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3869 00000100:00080000:0.0:1788572721.349387:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572721.349394:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788572721.349432:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488557312/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572721.349465:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572721.349468:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572721.349469:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572721.349473:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3869 00000100:00080000:0.0:1788572721.349476:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572721.349479:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572721.349487:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572721.349498:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488557440/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572721.349506:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572721.349509:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572721.349511:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572721.349514:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3869 00000100:00080000:0.0:1788572721.349516:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572721.349519:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572721.349522:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572721.349529:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488557568/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572721.349537:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572721.349542:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572721.349545:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788572721.349546:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488557312:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788572721.349553:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3869 00000100:00080000:0.0:1788572721.349556:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572721.349567:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572721.349575:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572721.349582:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488557696/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572721.349588:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572721.349591:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572721.349592:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572721.349598:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3869 00000100:00080000:0.0:1788572721.349600:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572721.349603:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572721.349606:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572721.349612:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488557824/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788572721.349669:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572721.349720:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572721/real 1788572721] req@ffff94d408d3ea00 x1875450488557312/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572721.349743:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488557312/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572721.349756:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572721.349762:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572721.349767:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488557312:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788572721.349778:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488557440:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572721.349807:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572721.349823:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572721/real 1788572721] req@ffff94d408d3ce00 x1875450488557440/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572721.349838:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488557440/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572721.349847:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572721.349851:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572721.349855:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488557440:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572721.349890:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488557568:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572721.349947:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572721.349962:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572721/real 1788572721] req@ffff94d408d3df80 x1875450488557568/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572721.349976:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488557568/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572721.349986:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572721.349989:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572721.349993:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488557568:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572721.350000:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488557696:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572721.350027:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572721.350041:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572721/real 1788572721] req@ffff94d408d3d180 x1875450488557696/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572721.350056:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488557696/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572721.350065:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572721.350069:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572721.350073:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488557696:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572721.350080:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488557824:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572721.350103:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572721.350116:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572721/real 1788572721] req@ffff94d408d3c000 x1875450488557824/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572721.350131:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488557824/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572721.350148:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572721.350152:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572721.350156:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488557824:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572721.861253:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572722.309379:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572722.885249:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572723.333387:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572723.909279:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572724.357341:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572724.933390:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572725.381405:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572725.957278:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572726.405332:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572726.469308:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572726.469319:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572726.469324:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572726.469343:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3874 00000100:00080000:0.0:1788572726.469356:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572726.469364:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788572726.469407:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488557952/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572726.469442:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572726.469446:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572726.469447:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572726.469450:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3874 00000100:00080000:0.0:1788572726.469452:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572726.469455:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572726.469464:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572726.469477:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488558080/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572726.469485:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572726.469489:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572726.469490:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572726.469492:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3874 00000100:00080000:0.0:1788572726.469494:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572726.469499:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572726.469507:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572726.469509:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488557952:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788572726.469518:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488558208/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572726.469533:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572726.469536:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572726.469537:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572726.469541:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3874 00000100:00080000:0.0:1788572726.469545:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572726.469550:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572726.469556:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572726.469571:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488558336/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572726.469582:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572726.469589:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1788572726.469589:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788572726.469598:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572726.469607:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3874 00000100:00080000:0.0:1788572726.469615:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572726.469622:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572726.469630:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572726.469637:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572726/real 1788572726] req@ffff94d408d3dc00 x1875450488557952/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572726.469644:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488558464/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572726.469684:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488557952/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572726.469696:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572726.469700:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572726.469703:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488557952:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788572726.469710:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488558080:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572726.469740:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572726.469759:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572726/real 1788572726] req@ffff94d408d3c380 x1875450488558080/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572726.469768:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488558080/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572726.469773:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572726.469776:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572726.469778:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488558080:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572726.469830:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488558208:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572726.469853:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572726.469866:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572726/real 1788572726] req@ffff94d408d3c700 x1875450488558208/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572726.469879:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488558208/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572726.469888:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572726.469892:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572726.469896:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488558208:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572726.469903:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488558336:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572726.469927:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572726.469939:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572726/real 1788572726] req@ffff94d408d3e300 x1875450488558336/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572726.469951:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488558336/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572726.469960:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572726.469963:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572726.469966:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488558336:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572726.469972:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488558464:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572726.469991:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572726.470002:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572726/real 1788572726] req@ffff94d408d3d880 x1875450488558464/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572726.470015:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488558464/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572726.470024:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572726.470027:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572726.470031:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488558464:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572726.981361:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572727.429352:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572728.005611:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572728.453348:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572729.029413:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572729.477411:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572730.053402:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572730.501314:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572731.077294:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572731.525323:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572731.589347:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572731.589360:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572731.589365:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572731.589377:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3879 00000100:00080000:2.0:1788572731.589390:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572731.589399:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788572731.589444:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488558592/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572731.589477:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572731.589480:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572731.589482:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572731.589485:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3879 00000100:00080000:2.0:1788572731.589488:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572731.589490:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572731.589500:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572731.589511:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488558720/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572731.589520:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788572731.589523:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488558592:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788572731.589523:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572731.589525:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572731.589533:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3879 00000100:00080000:2.0:1788572731.589536:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572731.589538:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572731.589542:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572731.589550:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488558848/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572731.589556:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572731.589558:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572731.589559:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572731.589563:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3879 00000100:00080000:2.0:1788572731.589565:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572731.589568:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572731.589571:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572731.589578:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488558976/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572731.589584:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572731.589587:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572731.589588:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572731.589590:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3879 00000100:00080000:2.0:1788572731.589594:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572731.589596:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572731.589599:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572731.589605:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488559104/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788572731.589631:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572731.589702:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572731/real 1788572731] req@ffff94d408c60a80 x1875450488558592/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572731.589726:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488558592/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572731.589739:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572731.589744:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572731.589749:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488558592:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788572731.589781:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488558720:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572731.589810:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572731.589825:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572731/real 1788572731] req@ffff94d408c62d80 x1875450488558720/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572731.589840:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488558720/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572731.589856:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572731.589860:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572731.589864:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488558720:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572731.589871:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488558848:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572731.589892:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572731.589905:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572731/real 1788572731] req@ffff94d408c60000 x1875450488558848/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572731.589918:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488558848/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572731.589927:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572731.589931:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572731.589934:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488558848:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572731.589941:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488558976:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572731.589967:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572731.589979:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572731/real 1788572731] req@ffff94d408c61180 x1875450488558976/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572731.589994:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488558976/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572731.590003:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572731.590007:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572731.590011:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488558976:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572731.590019:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488559104:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572731.590040:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572731.590052:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572731/real 1788572731] req@ffff94d408c62680 x1875450488559104/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572731.590067:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488559104/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572731.590076:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572731.590080:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572731.590083:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488559104:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572732.101275:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572732.549387:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572733.125331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572733.573389:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572734.149289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572734.597384:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572735.173263:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572735.621320:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572736.197275:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572736.645384:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572736.709307:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572736.709319:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572736.709324:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572736.709334:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3885 00000100:00080000:0.0:1788572736.709344:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572736.709350:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788572736.709382:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488559232/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572736.709408:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572736.709411:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572736.709412:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572736.709415:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3885 00000100:00080000:0.0:1788572736.709417:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572736.709420:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572736.709428:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572736.709437:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ed80 x1875450488559360/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572736.709444:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572736.709447:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572736.709449:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572736.709451:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3885 00000100:00080000:0.0:1788572736.709454:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572736.709456:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572736.709459:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572736.709465:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488559232:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788572736.709472:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488559488/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572736.709480:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572736.709483:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572736.709484:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572736.709487:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3885 00000100:00080000:0.0:1788572736.709489:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572736.709491:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572736.709494:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572736.709502:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488559616/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572736.709509:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572736.709511:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572736.709513:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572736.709524:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3885 00000800:00000100:1.0:1788572736.709524:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788572736.709542:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572736.709547:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572736.709552:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572736.709563:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488559744/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572736.709582:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572736/real 1788572736] req@ffff94d408d3f480 x1875450488559232/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572736.709604:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488559232/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572736.709616:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572736.709620:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572736.709624:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488559232:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788572736.709633:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488559360:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572736.709656:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572736.709677:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572736/real 1788572736] req@ffff94d408d3ed80 x1875450488559360/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572736.709687:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ed80 x1875450488559360/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572736.709693:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572736.709695:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572736.709698:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488559360:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572736.709717:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488559488:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572736.709730:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572736.709737:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572736/real 1788572736] req@ffff94d408d3d880 x1875450488559488/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572736.709745:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488559488/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572736.709751:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572736.709753:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572736.709755:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488559488:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572736.709759:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488559616:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572736.709769:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572736.709774:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572736/real 1788572736] req@ffff94d408d3e300 x1875450488559616/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572736.709781:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488559616/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572736.709787:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572736.709789:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572736.709791:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488559616:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572736.709794:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488559744:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572736.709803:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572736.709808:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572736/real 1788572736] req@ffff94d408d3c700 x1875450488559744/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572736.709815:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488559744/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572736.709820:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572736.709823:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572736.709827:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488559744:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572737.221263:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572737.669358:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572738.245307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572738.693375:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572739.269270:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572739.717402:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572740.293291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572740.741379:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572741.317259:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572741.765460:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572741.829341:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572741.829353:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572741.829358:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572741.829368:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3890 00000100:00080000:0.0:1788572741.829392:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572741.829416:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788572741.829462:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488559872/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572741.829508:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572741.829512:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572741.829514:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572741.829520:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3890 00000100:00080000:0.0:1788572741.829525:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572741.829531:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572741.829543:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572741.829549:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488559872:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788572741.829568:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488560000/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572741.829594:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572741.829602:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572741.829605:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572741.829612:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3890 00000100:00080000:0.0:1788572741.829618:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572741.829623:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572741.829631:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572741.829652:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488560128/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572741.829667:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788572741.829670:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788572741.829676:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572741.829683:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572741.829694:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3890 00000100:00080000:0.0:1788572741.829703:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572741.829709:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572741.829719:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572741.829737:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488560256/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572741.829748:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572741.829751:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572741.829753:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572741.829759:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3890 00000100:00080000:0.0:1788572741.829765:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572741.829786:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788572741.829797:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572741/real 1788572741] req@ffff94d408d3c380 x1875450488559872/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788572741.829798:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572741.829818:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488560384/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572741.830056:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488559872/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572741.830072:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572741.830078:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572741.830083:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488559872:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788572741.830131:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488560000:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788572741.830174:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572741.830191:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572741/real 1788572741] req@ffff94d408d3dc00 x1875450488560000/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572741.830240:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488560000/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572741.830252:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572741.830258:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572741.830266:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488560000:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788572741.830276:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488560128:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788572741.830302:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572741.830317:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572741/real 1788572741] req@ffff94d408d3c000 x1875450488560128/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572741.830332:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488560128/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572741.830343:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572741.830347:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572741.830351:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488560128:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788572741.830359:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488560256:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788572741.830383:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572741.830397:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572741/real 1788572741] req@ffff94d408d3d180 x1875450488560256/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572741.830442:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488560256/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572741.830452:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572741.830456:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572741.830460:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488560256:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572741.830468:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488560384:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788572741.830491:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572741.830506:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572741/real 1788572741] req@ffff94d408d3df80 x1875450488560384/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572741.830519:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488560384/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572741.830528:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572741.830531:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572741.830535:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488560384:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572742.341304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572742.789413:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572743.365285:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572743.813362:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572744.389277:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572744.837399:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572745.413286:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572745.861329:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572746.437338:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572746.885426:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572746.949359:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572746.949374:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572746.949381:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572746.949395:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3895 00000100:00080000:0.0:1788572746.949420:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572746.949431:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788572746.949485:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488560512/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572746.949515:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572746.949519:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572746.949520:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572746.949523:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3895 00000100:00080000:0.0:1788572746.949526:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572746.949530:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572746.949541:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572746.949554:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488560640/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572746.949562:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572746.949568:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572746.949572:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788572746.949572:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488560512:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788572746.949588:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3895 00000100:00080000:0.0:1788572746.949593:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572746.949596:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572746.949602:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572746.949616:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488560768/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572746.949630:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572746.949633:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572746.949634:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572746.949637:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3895 00000100:00080000:0.0:1788572746.949640:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572746.949643:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572746.949647:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572746.949655:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f100 x1875450488560896/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572746.949662:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572746.949672:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1788572746.949672:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788572746.949678:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572746.949686:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3895 00000100:00080000:0.0:1788572746.949691:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572746.949695:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572746.949700:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572746.949707:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488561024/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572746.949735:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572746/real 1788572746] req@ffff94d408d3ce00 x1875450488560512/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572746.949759:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488560512/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572746.949771:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572746.949776:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572746.949780:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488560512:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788572746.949811:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488560640:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788572746.949840:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572746.949863:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572746/real 1788572746] req@ffff94d408d3ea00 x1875450488560640/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572746.949878:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488560640/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572746.949888:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572746.949891:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572746.949895:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488560640:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788572746.949902:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488560768:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788572746.949924:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572746.949934:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572746/real 1788572746] req@ffff94d408d3e680 x1875450488560768/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572746.949943:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488560768/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572746.949951:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572746.949955:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572746.949958:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488560768:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788572746.949965:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488560896:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788572746.949988:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572746.950001:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572746/real 1788572746] req@ffff94d408d3f100 x1875450488560896/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572746.950012:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f100 x1875450488560896/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572746.950018:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572746.950020:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572746.950022:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488560896:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572746.950027:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488561024:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788572746.950041:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572746.950048:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572746/real 1788572746] req@ffff94d408d3ca80 x1875450488561024/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572746.950055:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488561024/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572746.950060:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572746.950062:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572746.950064:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488561024:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572747.461359:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572747.909407:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572748.485338:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788572748.933438:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572749.509293:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572749.957407:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572750.533320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572750.981427:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572751.557295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572752.005362:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572752.069348:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572752.069358:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572752.069362:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572752.069371:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3900 00000100:00080000:3.0:1788572752.069379:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572752.069386:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788572752.069418:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488561152/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572752.069444:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572752.069446:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572752.069448:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572752.069451:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3900 00000100:00080000:3.0:1788572752.069453:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572752.069455:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572752.069462:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572752.069470:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488561280/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572752.069477:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572752.069480:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572752.069482:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572752.069485:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3900 00000100:00080000:3.0:1788572752.069487:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572752.069489:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572752.069492:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572752.069499:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488561408/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572752.069510:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488561152:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788572752.069513:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572752.069518:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572752.069522:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572752.069525:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3900 00000100:00080000:3.0:1788572752.069527:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572752.069529:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572752.069532:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572752.069542:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488561536/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572752.069548:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572752.069550:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572752.069552:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572752.069554:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3900 00000100:00080000:3.0:1788572752.069557:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572752.069559:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572752.069562:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572752.069568:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488561664/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572752.069622:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488561280:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572752.069663:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488561408:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572752.069695:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488561536:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572752.069723:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488561664:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572752.581267:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572753.029352:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572753.605395:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572754.053391:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572754.629257:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572755.077376:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788572755.141516:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:2.0:1788572755.141553:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788572755.141575:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788572755.141615:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788572755.141625:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788572755.141636:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788572755.141651:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:1.0:1788572755.141708:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572752/real 1788572755] req@ffff94d408c54700 x1875450488561408/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572755.141751:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488561408/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572755.141766:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572755.141773:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572755.141778:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488561408:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572755.141789:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572752/real 1788572755] req@ffff94d408c55f80 x1875450488561536/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572755.141804:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488561536/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572755.141818:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572755.141822:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572755.141825:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488561536:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572755.141833:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572752/real 1788572755] req@ffff94d408c56300 x1875450488561664/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572755.141845:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488561664/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572755.141853:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572755.141858:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572755.141862:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488561664:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572755.141873:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572752/real 1788572755] req@ffff94d408c57480 x1875450488561152/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572755.141891:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488561152/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572755.141899:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572755.141903:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572755.141907:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488561152:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788572755.141915:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572752/real 1788572755] req@ffff94d408c54000 x1875450488561280/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572755.141931:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488561280/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572755.141941:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572755.141946:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572755.141949:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488561280:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788572755.653330:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572756.101418:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572756.677275:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572757.125376:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572757.189437:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572757.189448:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572757.189453:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572757.189464:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3905 00000100:00080000:2.0:1788572757.189475:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572757.189483:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788572757.189519:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60700 x1875450488561792/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572757.189559:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572757.189563:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572757.189565:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572757.189570:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3905 00000100:00080000:2.0:1788572757.189574:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572757.189578:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572757.189588:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572757.189602:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60380 x1875450488561920/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572757.189614:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572757.189617:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572757.189620:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572757.189624:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3905 00000100:00080000:2.0:1788572757.189627:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572757.189631:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572757.189636:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572757.189648:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62300 x1875450488562048/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572757.189657:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572757.189660:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572757.189662:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572757.189666:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3905 00000100:00080000:2.0:1788572757.189669:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572757.189672:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572757.189677:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572757.189688:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488562176/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572757.189698:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572757.189701:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572757.189703:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572757.189707:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3905 00000100:00080000:2.0:1788572757.189710:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572757.189714:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572757.189719:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572757.189730:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488562304/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572757.189852:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488561792:192.168.202.125@tcp:250:kworker.0 00000800:00000100:1.0:1788572757.189943:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572757.189982:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572757/real 1788572757] req@ffff94d408c60700 x1875450488561792/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572757.190002:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60700 x1875450488561792/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572757.190022:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572757.190026:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572757.190031:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488561792:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788572757.190039:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488561920:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572757.190065:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572757.190080:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572757/real 1788572757] req@ffff94d408c60380 x1875450488561920/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572757.190101:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60380 x1875450488561920/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572757.190109:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572757.190114:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572757.190116:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488561920:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572757.190123:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488562048:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572757.190143:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572757.190173:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572757/real 1788572757] req@ffff94d408c62300 x1875450488562048/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572757.190184:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62300 x1875450488562048/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572757.190192:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572757.190195:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572757.190198:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488562048:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572757.190236:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488562176:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572757.190273:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572757.190286:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572757/real 1788572757] req@ffff94d408c62680 x1875450488562176/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572757.190298:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488562176/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572757.190306:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572757.190309:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572757.190313:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488562176:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572757.190318:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488562304:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572757.190339:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572757.190351:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572757/real 1788572757] req@ffff94d408c61180 x1875450488562304/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572757.190364:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488562304/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572757.190373:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572757.190377:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572757.190380:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488562304:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572757.701331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572758.149400:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572758.725352:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572759.173419:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572759.749373:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572760.197432:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572760.773637:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572761.221425:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572761.797339:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572762.245424:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572762.309338:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572762.309350:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572762.309355:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572762.309366:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3910 00000100:00080000:2.0:1788572762.309377:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572762.309385:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788572762.309425:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488562432/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572762.309447:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572762.309449:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572762.309451:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572762.309453:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3910 00000100:00080000:2.0:1788572762.309456:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572762.309459:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572762.309466:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572762.309477:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488562560/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572762.309490:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572762.309494:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572762.309496:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572762.309500:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3910 00000100:00080000:2.0:1788572762.309504:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572762.309510:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572762.309515:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572762.309529:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488562432:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788572762.309530:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488562688/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572762.309570:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572762.309573:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572762.309574:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572762.309579:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3910 00000100:00080000:2.0:1788572762.309583:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572762.309588:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572762.309592:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572762.309601:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488562816/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572762.309608:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572762.309611:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572762.309612:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572762.309614:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3910 00000100:00080000:2.0:1788572762.309616:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572762.309619:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572762.309622:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572762.309628:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488562944/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788572762.309657:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572762.309709:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572762/real 1788572762] req@ffff94d408c60000 x1875450488562432/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572762.309734:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488562432/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572762.309748:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572762.309753:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572762.309758:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488562432:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788572762.309768:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488562560:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572762.309797:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572762.309813:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572762/real 1788572762] req@ffff94d408c62d80 x1875450488562560/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572762.309827:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488562560/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572762.309835:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572762.309839:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572762.309842:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488562560:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572762.309873:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488562688:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572762.309897:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572762.309910:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572762/real 1788572762] req@ffff94d408c60a80 x1875450488562688/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572762.309932:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488562688/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572762.309941:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572762.309945:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572762.309948:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488562688:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572762.309955:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488562816:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572762.309986:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572762.309999:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572762/real 1788572762] req@ffff94d408c63800 x1875450488562816/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572762.310012:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488562816/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572762.310021:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572762.310025:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572762.310028:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488562816:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572762.310036:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488562944:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572762.310058:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572762.310070:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572762/real 1788572762] req@ffff94d408c62a00 x1875450488562944/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572762.310083:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488562944/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572762.310099:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572762.310103:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572762.310106:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488562944:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572762.821713:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572763.269409:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572763.845363:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572764.293401:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572764.871114:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572765.317374:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572765.893388:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572766.341423:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572766.917310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572767.365372:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572767.429313:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572767.429328:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572767.429332:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572767.429343:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3915 00000100:00080000:2.0:1788572767.429354:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572767.429361:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788572767.429399:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488563072/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572767.429433:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572767.429437:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572767.429440:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572767.429445:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3915 00000100:00080000:2.0:1788572767.429449:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572767.429454:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572767.429464:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572767.429480:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488563200/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572767.429483:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488563072:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788572767.429499:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572767.429502:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572767.429504:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572767.429508:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3915 00000100:00080000:2.0:1788572767.429511:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572767.429513:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572767.429518:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572767.429528:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488563328/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572767.429536:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572767.429538:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572767.429539:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572767.429542:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3915 00000100:00080000:2.0:1788572767.429545:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788572767.429547:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788572767.429551:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572767.429556:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572767.429565:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60e00 x1875450488563456/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572767.429571:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572767.429574:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572767.429575:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572767.429579:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3915 00000100:00080000:2.0:1788572767.429582:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788572767.429586:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572767/real 1788572767] req@ffff94d408c63b80 x1875450488563072/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572767.429587:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572767.429593:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572767.429605:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63100 x1875450488563584/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572767.429610:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488563072/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572767.429623:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572767.429626:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572767.429630:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488563072:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788572767.429649:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488563200:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572767.429665:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572767.429673:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572767/real 1788572767] req@ffff94d408c61500 x1875450488563200/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572767.429681:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488563200/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572767.429691:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572767.429693:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572767.429695:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488563200:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572767.429698:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488563328:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572767.429709:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572767.429714:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572767/real 1788572767] req@ffff94d408c61c00 x1875450488563328/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572767.429721:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488563328/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572767.429726:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572767.429728:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572767.429730:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488563328:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572767.429733:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488563456:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572767.429743:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572767.429749:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572767/real 1788572767] req@ffff94d408c60e00 x1875450488563456/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572767.429756:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60e00 x1875450488563456/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572767.429761:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572767.429763:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572767.429765:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488563456:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572767.429769:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488563584:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572767.429778:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572767.429783:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572767/real 1788572767] req@ffff94d408c63100 x1875450488563584/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572767.429790:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63100 x1875450488563584/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572767.429795:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572767.429797:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572767.429798:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488563584:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572767.941323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572768.389389:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572768.965339:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572769.413404:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572769.989349:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572770.437440:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572771.013300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572771.461393:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572772.037377:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572772.485397:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572772.549405:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572772.549417:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572772.549423:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572772.549437:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3920 00000100:00080000:3.0:1788572772.549452:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572772.549462:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788572772.549513:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488563712/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572772.549547:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572772.549551:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572772.549554:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572772.549558:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3920 00000100:00080000:3.0:1788572772.549563:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572772.549569:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572772.549581:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572772.549595:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488563712:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788572772.549598:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488563840/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572772.549613:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572772.549617:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572772.549620:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572772.549624:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3920 00000100:00080000:3.0:1788572772.549628:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572772.549633:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572772.549639:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572772.549654:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488563968/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572772.549665:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572772.549669:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572772.549671:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788572772.549678:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788572772.549679:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3920 00000100:00080000:3.0:1788572772.549685:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572772.549690:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572772.549697:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572772.549711:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488564096/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572772.549722:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572772/real 1788572772] req@ffff94d408c55180 x1875450488563712/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572772.549722:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572772.549731:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572772.549736:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572772.549743:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3920 00000100:00080000:3.0:1788572772.549747:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788572772.549748:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488563712/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572772.549757:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00080000:1.0:1788572772.549763:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788572772.549765:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788572772.549770:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572772.549775:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488563712:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788572772.549783:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488564224/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572772.549812:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488563840:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572772.549833:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572772.549844:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572772/real 1788572772] req@ffff94d408c56a00 x1875450488563840/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572772.549853:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488563840/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572772.549859:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572772.549861:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572772.549863:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488563840:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572772.549867:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488563968:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572772.549878:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572772.549884:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572772/real 1788572772] req@ffff94d408c55880 x1875450488563968/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572772.549892:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488563968/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572772.549897:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572772.549899:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572772.549901:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488563968:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572772.549904:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488564096:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572772.549915:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572772.549921:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572772/real 1788572772] req@ffff94d408c54380 x1875450488564096/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572772.549933:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488564096/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572772.549938:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572772.549940:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572772.549942:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488564096:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572772.549945:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488564224:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572772.549955:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572772.549960:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572772/real 1788572772] req@ffff94d408c57800 x1875450488564224/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572772.549968:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488564224/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572772.549973:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572772.549974:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572772.549976:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488564224:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572773.061333:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572773.509406:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572774.085297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572774.533429:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572775.109306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572775.557407:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572776.133294:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572776.581369:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572777.157331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572777.605389:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572777.669302:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572777.669310:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572777.669314:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572777.669322:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3925 00000100:00080000:3.0:1788572777.669331:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572777.669339:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788572777.669375:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488564352/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572777.669403:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572777.669408:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572777.669410:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572777.669414:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3925 00000100:00080000:3.0:1788572777.669418:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572777.669424:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572777.669434:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572777.669449:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488564352:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788572777.669450:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488564480/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572777.669465:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572777.669468:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572777.669470:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572777.669474:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3925 00000100:00080000:3.0:1788572777.669477:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572777.669481:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572777.669484:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572777.669495:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488564608/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788572777.669513:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572777.669545:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572777/real 1788572777] req@ffff94d408c56d80 x1875450488564352/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572777.669562:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488564352/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572777.669571:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572777.669574:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:3.0:1788572777.669575:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788572777.669580:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488564352:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788572777.669581:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572777.669585:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572777.669591:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3925 00000100:00100000:0.0:1788572777.669606:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488564480:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572777.669628:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572777.669638:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572777/real 1788572777] req@ffff94d408c57100 x1875450488564480/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572777.669652:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488564480/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572777.669660:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572777.669664:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572777.669669:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488564480:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572777.669676:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488564608:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788572777.669685:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572777.669691:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572777.669697:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788572777.669708:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572777.669721:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572777/real 1788572777] req@ffff94d408c55c00 x1875450488564608/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572777.669732:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488564608/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572777.669740:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572777.669743:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572777.669751:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488564608:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788572777.669751:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488564736/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572777.669769:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572777.669772:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572777.669773:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788572777.669777:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488564736:192.168.202.125@tcp:8:kworker.0 00000100:00080000:3.0:1788572777.669779:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3925 00000100:00080000:3.0:1788572777.669785:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572777.669789:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572777.669793:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572777.669804:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488564864/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788572777.669813:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572777.669829:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572777/real 1788572777] req@ffff94d408c54e00 x1875450488564736/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572777.669839:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488564736/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572777.669850:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572777.669852:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572777.669854:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488564736:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572777.669863:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488564864:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572777.669877:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572777.669895:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572777/real 1788572777] req@ffff94d408c57b80 x1875450488564864/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572777.669903:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488564864/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572777.669909:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572777.669911:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572777.669913:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488564864:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572778.181266:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572778.629339:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572779.205260:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572779.653400:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572780.229362:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572780.677400:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572781.253323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572781.701476:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572782.277352:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572782.725355:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572782.789275:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572782.789300:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572782.789304:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572782.789313:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3931 00000100:00080000:3.0:1788572782.789321:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572782.789327:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788572782.789359:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488564992/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572782.789386:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572782.789389:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572782.789393:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572782.789396:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3931 00000100:00080000:3.0:1788572782.789398:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572782.789403:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572782.789413:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572782.789426:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55500 x1875450488565120/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572782.789433:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488564992:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788572782.789444:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572782.789452:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572782.789454:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572782.789458:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3931 00000100:00080000:3.0:1788572782.789462:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572782.789465:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572782.789469:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572782.789481:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488565248/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572782.789489:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572782.789491:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572782.789493:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572782.789495:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3931 00000100:00080000:3.0:1788572782.789498:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572782.789500:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572782.789504:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788572782.789508:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572782.789515:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488565376/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572782.789522:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572782.789525:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572782.789527:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572782.789529:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3931 00000100:00080000:3.0:1788572782.789532:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572782.789535:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572782.789538:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572782.789546:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488565504/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572782.789554:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572782/real 1788572782] req@ffff94d408c56680 x1875450488564992/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572782.789576:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488564992/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572782.789588:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572782.789592:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572782.789597:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488564992:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788572782.789620:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488565120:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572782.789644:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572782.789657:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572782/real 1788572782] req@ffff94d408c55500 x1875450488565120/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572782.789671:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55500 x1875450488565120/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572782.789680:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572782.789684:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572782.789687:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488565120:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572782.789694:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488565248:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572782.789714:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572782.789727:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572782/real 1788572782] req@ffff94d408c54a80 x1875450488565248/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572782.789740:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488565248/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572782.789756:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572782.789760:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572782.789763:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488565248:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572782.789770:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488565376:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572782.789791:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572782.789803:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572782/real 1788572782] req@ffff94d408c57b80 x1875450488565376/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572782.789816:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488565376/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572782.789825:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572782.789828:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572782.789832:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488565376:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572782.789849:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488565504:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572782.789868:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572782.789879:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572782/real 1788572782] req@ffff94d408c54e00 x1875450488565504/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572782.789892:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488565504/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572782.789900:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572782.789904:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572782.789908:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488565504:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572783.301348:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572783.749379:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572784.325329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572784.773361:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572785.349337:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572785.797399:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572786.373336:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572786.821384:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572787.397318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572787.845414:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572787.909303:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572787.909312:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572787.909316:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572787.909324:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3936 00000100:00080000:2.0:1788572787.909334:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572787.909340:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788572787.909373:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488565632/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572787.909394:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572787.909396:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572787.909398:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572787.909401:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3936 00000100:00080000:2.0:1788572787.909403:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572787.909405:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572787.909413:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572787.909422:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488565760/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572787.909431:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788572787.909432:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488565632:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788572787.909437:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572787.909440:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572787.909443:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3936 00000100:00080000:2.0:1788572787.909445:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572787.909448:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572787.909451:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572787.909459:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488565888/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572787.909466:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572787.909469:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572787.909471:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572787.909474:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3936 00000100:00080000:2.0:1788572787.909477:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572787.909479:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572787.909483:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572787.909491:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63100 x1875450488566016/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788572787.909495:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788572787.909504:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572787.909507:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572787.909508:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572787.909512:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3936 00000100:00080000:2.0:1788572787.909515:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572787.909518:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572787.909521:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572787.909530:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572787/real 1788572787] req@ffff94d408c63480 x1875450488565632/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572787.909531:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60e00 x1875450488566144/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572787.909571:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488565632/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572787.909580:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572787.909583:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572787.909586:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488565632:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788572787.909604:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488565760:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572787.909619:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572787.909627:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572787/real 1788572787] req@ffff94d408c61880 x1875450488565760/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572787.909635:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488565760/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572787.909640:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572787.909643:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572787.909645:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488565760:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572787.909649:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488565888:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572787.909659:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572787.909665:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572787/real 1788572787] req@ffff94d408c61f80 x1875450488565888/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572787.909672:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488565888/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572787.909677:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572787.909679:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572787.909681:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488565888:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572787.909684:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488566016:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572787.909694:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572787.909700:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572787/real 1788572787] req@ffff94d408c63100 x1875450488566016/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572787.909707:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63100 x1875450488566016/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572787.909712:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572787.909714:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572787.909716:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488566016:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572787.909719:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488566144:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572787.909728:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572787.909733:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572787/real 1788572787] req@ffff94d408c60e00 x1875450488566144/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572787.909740:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60e00 x1875450488566144/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572787.909745:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572787.909747:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572787.909749:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488566144:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572788.421304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572788.869378:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572789.445352:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572789.893423:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572790.469283:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572790.917332:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572791.493327:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572791.941399:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572792.517316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572792.965412:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572793.029345:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572793.029356:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572793.029362:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572793.029372:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3941 00000100:00080000:2.0:1788572793.029385:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572793.029393:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788572793.029435:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488566272/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572793.029463:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572793.029467:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572793.029469:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572793.029473:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3941 00000100:00080000:2.0:1788572793.029477:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572793.029482:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572793.029491:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572793.029503:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488566272:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788572793.029507:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488566400/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572793.029521:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572793.029525:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572793.029528:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572793.029531:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3941 00000100:00080000:2.0:1788572793.029535:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572793.029539:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572793.029544:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572793.029556:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488566528/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572793.029565:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572793.029569:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1788572793.029577:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788572793.029581:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572793.029588:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3941 00000100:00080000:2.0:1788572793.029593:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572793.029597:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572793.029603:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572793.029615:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488566656/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572793.029620:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572793/real 1788572793] req@ffff94d408c61c00 x1875450488566272/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572793.029630:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572793.029639:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572793.029643:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572793.029648:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3941 00000100:00080000:2.0:1788572793.029652:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788572793.029655:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488566272/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572793.029659:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572793.029668:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788572793.029673:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572793.029681:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788572793.029683:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488566784/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572793.029690:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488566272:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788572793.029723:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488566400:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572793.029751:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572793.029766:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572793/real 1788572793] req@ffff94d408c61500 x1875450488566400/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572793.029781:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488566400/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572793.029790:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572793.029794:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572793.029797:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488566400:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572793.029804:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488566528:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572793.029824:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572793.029834:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572793/real 1788572793] req@ffff94d408c63b80 x1875450488566528/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572793.029846:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488566528/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572793.029855:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572793.029859:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572793.029863:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488566528:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572793.029869:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488566656:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572793.029887:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572793.029897:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572793/real 1788572793] req@ffff94d408c62a00 x1875450488566656/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572793.029916:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488566656/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572793.029925:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572793.029928:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572793.029932:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488566656:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572793.029937:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488566784:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572793.029954:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572793.029964:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572793/real 1788572793] req@ffff94d408c63800 x1875450488566784/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572793.029976:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488566784/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572793.029984:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572793.029988:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572793.029991:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488566784:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572793.541348:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572793.989391:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572794.565385:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572795.013402:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572795.589276:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572796.037338:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572796.613357:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572797.061408:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572797.637280:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572798.085336:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572798.149338:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572798.149347:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572798.149351:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572798.149360:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3946 00000100:00080000:2.0:1788572798.149368:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572798.149375:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788572798.149417:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488566912/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572798.149447:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572798.149453:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572798.149454:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572798.149457:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3946 00000100:00080000:2.0:1788572798.149460:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572798.149464:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572798.149471:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572798.149482:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488567040/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572798.149491:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572798.149493:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572798.149494:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572798.149497:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3946 00000100:00080000:2.0:1788572798.149499:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572798.149501:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572798.149504:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572798.149510:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488566912:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788572798.149514:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488567168/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572798.149525:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572798.149527:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572798.149528:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572798.149531:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3946 00000100:00080000:2.0:1788572798.149533:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572798.149535:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572798.149540:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572798.149547:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488567296/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572798.149553:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572798.149556:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572798.149557:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572798.149559:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3946 00000100:00080000:2.0:1788572798.149562:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572798.149564:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572798.149567:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572798.149573:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488567424/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788572798.149585:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572798.149639:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572798/real 1788572798] req@ffff94d408c60a80 x1875450488566912/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572798.149667:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488566912/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572798.149679:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572798.149684:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572798.149688:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488566912:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788572798.149696:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488567040:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572798.149721:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572798.149737:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572798/real 1788572798] req@ffff94d408c62d80 x1875450488567040/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572798.149750:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488567040/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572798.149759:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572798.149762:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572798.149766:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488567040:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572798.149791:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488567168:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572798.149815:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572798.149828:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572798/real 1788572798] req@ffff94d408c60000 x1875450488567168/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572798.149841:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488567168/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572798.149856:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572798.149859:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572798.149862:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488567168:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572798.149868:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488567296:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572798.149891:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572798.149904:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572798/real 1788572798] req@ffff94d408c61180 x1875450488567296/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572798.149916:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488567296/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572798.149936:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572798.149940:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572798.149944:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488567296:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572798.149951:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488567424:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572798.149972:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572798.149984:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572798/real 1788572798] req@ffff94d408c62680 x1875450488567424/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572798.149996:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488567424/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572798.150004:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572798.150008:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572798.150012:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488567424:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572798.661364:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572799.109466:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572799.685346:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572800.133375:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572800.709291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572801.157440:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572801.733360:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572802.181398:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572802.757358:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572803.205427:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572803.269411:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572803.269425:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572803.269431:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572803.269449:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3951 00000100:00080000:2.0:1788572803.269463:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572803.269473:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788572803.269529:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62300 x1875450488567552/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572803.269558:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572803.269561:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572803.269562:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572803.269566:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3951 00000100:00080000:2.0:1788572803.269571:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572803.269576:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572803.269587:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572803.269601:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60380 x1875450488567680/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572803.269620:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788572803.269624:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488567552:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788572803.269626:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572803.269630:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572803.269638:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3951 00000100:00080000:2.0:1788572803.269642:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572803.269645:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572803.269650:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572803.269663:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60700 x1875450488567808/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572803.269679:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572803.269683:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572803.269684:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572803.269689:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3951 00000100:00080000:2.0:1788572803.269696:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572803.269700:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572803.269705:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572803.269717:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488567936/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788572803.269725:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788572803.269733:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572803.269736:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572803.269737:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572803.269740:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3951 00000100:00080000:2.0:1788572803.269743:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572803.269748:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572803.269753:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572803.269773:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488568064/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572803.269779:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572803/real 1788572803] req@ffff94d408c62300 x1875450488567552/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572803.269802:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62300 x1875450488567552/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572803.269814:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572803.269819:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572803.269823:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488567552:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788572803.269848:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488567680:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572803.269870:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572803.269882:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572803/real 1788572803] req@ffff94d408c60380 x1875450488567680/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572803.269893:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60380 x1875450488567680/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572803.269900:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572803.269903:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572803.269907:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488567680:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572803.269912:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488567808:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572803.269927:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572803.269936:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572803/real 1788572803] req@ffff94d408c60700 x1875450488567808/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572803.269947:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60700 x1875450488567808/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572803.269955:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572803.269958:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572803.269960:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488567808:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572803.269966:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488567936:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572803.269982:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572803.269992:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572803/real 1788572803] req@ffff94d408c62680 x1875450488567936/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572803.270003:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488567936/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572803.270010:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572803.270013:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572803.270016:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488567936:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572803.270021:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488568064:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572803.270038:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572803.270048:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572803/real 1788572803] req@ffff94d408c61180 x1875450488568064/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572803.270058:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488568064/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572803.270066:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572803.270069:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572803.270073:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488568064:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572803.781339:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572804.229447:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572804.805282:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572805.253382:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572805.829296:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572806.277422:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572806.853283:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572807.301354:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572807.877261:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572808.325410:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572808.389359:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572808.389371:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572808.389376:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572808.389387:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3956 00000100:00080000:2.0:1788572808.389399:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572808.389407:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788572808.389454:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488568192/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572808.389493:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572808.389498:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572808.389500:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572808.389504:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3956 00000100:00080000:2.0:1788572808.389509:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572808.389514:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572808.389527:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572808.389543:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488568320/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572808.389552:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488568192:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788572808.389557:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572808.389561:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572808.389563:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572808.389567:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3956 00000100:00080000:2.0:1788572808.389571:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572808.389574:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572808.389579:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572808.389587:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488568448/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572808.389594:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572808.389597:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572808.389598:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572808.389601:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3956 00000100:00080000:2.0:1788572808.389604:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572808.389606:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572808.389610:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572808.389616:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488568576/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572808.389622:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572808.389625:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572808.389627:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572808.389629:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3956 00000100:00080000:2.0:1788572808.389632:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572808.389635:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572808.389639:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788572808.389644:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788572808.389650:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488568704/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572808.389690:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572808/real 1788572808] req@ffff94d408c60000 x1875450488568192/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572808.389706:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488568192/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572808.389715:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572808.389718:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572808.389721:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488568192:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788572808.389744:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488568320:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572808.389764:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572808.389772:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572808/real 1788572808] req@ffff94d408c62d80 x1875450488568320/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572808.389780:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488568320/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572808.389786:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572808.389789:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572808.389792:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488568320:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572808.389799:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488568448:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572808.389819:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572808.389832:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572808/real 1788572808] req@ffff94d408c60a80 x1875450488568448/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572808.389846:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488568448/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572808.389855:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572808.389858:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572808.389862:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488568448:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572808.389869:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488568576:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572808.389891:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572808.389902:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572808/real 1788572808] req@ffff94d408c63800 x1875450488568576/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572808.389910:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488568576/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572808.389916:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572808.389918:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572808.389920:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488568576:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572808.389925:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488568704:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572808.389946:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572808.389954:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572808/real 1788572808] req@ffff94d408c62a00 x1875450488568704/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572808.389962:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488568704/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572808.389968:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572808.389970:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572808.389972:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488568704:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572808.901355:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572809.349459:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572809.925291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572810.373450:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572810.949290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572811.397388:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572811.973294:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572812.421413:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572812.997320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572813.445396:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572813.509327:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572813.509336:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572813.509362:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572813.509370:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3961 00000100:00080000:2.0:1788572813.509379:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572813.509385:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788572813.509425:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488568832/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572813.509455:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572813.509458:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572813.509460:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572813.509463:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3961 00000100:00080000:2.0:1788572813.509467:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572813.509470:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572813.509477:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572813.509487:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488568960/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572813.509499:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572813.509502:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572813.509504:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572813.509507:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3961 00000100:00080000:2.0:1788572813.509509:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572813.509511:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572813.509515:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572813.509524:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488569088/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572813.509532:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572813.509535:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572813.509536:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572813.509538:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3961 00000100:00080000:2.0:1788572813.509541:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572813.509549:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788572813.509550:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488568832:192.168.202.125@tcp:250:kworker.0 00000008:00100000:2.0:1788572813.509553:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572813.509571:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60e00 x1875450488569216/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572813.509579:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572813.509581:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572813.509582:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572813.509587:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3961 00000100:00080000:2.0:1788572813.509590:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572813.509592:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572813.509596:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572813.509610:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63100 x1875450488569344/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788572813.509650:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572813.509697:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572813/real 1788572813] req@ffff94d408c63b80 x1875450488568832/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572813.509718:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488568832/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572813.509729:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572813.509733:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572813.509736:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488568832:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788572813.509744:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488568960:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572813.509764:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572813.509774:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572813/real 1788572813] req@ffff94d408c61500 x1875450488568960/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572813.509800:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488568960/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572813.509819:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572813.509824:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572813.509829:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488568960:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572813.509864:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488569088:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572813.509888:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572813.509897:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572813/real 1788572813] req@ffff94d408c61c00 x1875450488569088/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572813.509912:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488569088/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572813.509923:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572813.509926:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572813.509929:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488569088:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572813.509938:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488569216:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572813.509966:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572813.509991:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572813/real 1788572813] req@ffff94d408c60e00 x1875450488569216/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572813.510000:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60e00 x1875450488569216/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572813.510005:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572813.510008:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572813.510012:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488569216:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572813.510019:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488569344:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572813.510042:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572813.510057:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572813/real 1788572813] req@ffff94d408c63100 x1875450488569344/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572813.510076:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63100 x1875450488569344/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572813.510089:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572813.510093:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572813.510096:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488569344:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572814.021320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572814.469426:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572815.045353:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572815.493420:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572816.069327:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572816.517451:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572817.093276:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572817.541445:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572818.117283:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572818.565490:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572818.629341:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572818.629350:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572818.629353:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572818.629361:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3966 00000100:00080000:3.0:1788572818.629371:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572818.629384:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788572818.629421:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488569472/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572818.629457:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572818.629461:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572818.629462:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572818.629466:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3966 00000100:00080000:3.0:1788572818.629468:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572818.629471:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572818.629479:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572818.629495:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488569600/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572818.629505:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572818.629508:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572818.629510:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572818.629519:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3966 00000100:00100000:0.0:1788572818.629526:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488569472:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788572818.629526:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572818.629545:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572818.629553:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572818.629577:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488569728/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572818.629587:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572818.629590:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572818.629591:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572818.629597:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3966 00000100:00080000:3.0:1788572818.629600:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572818.629604:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572818.629611:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572818.629621:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488569856/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572818.629628:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572818.629630:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572818.629631:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572818.629634:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3966 00000100:00080000:3.0:1788572818.629637:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572818.629641:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572818.629646:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572818.629658:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488569984/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572818.629671:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488569600:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572818.629704:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488569728:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572818.629725:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488569856:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572818.629744:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488569984:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572819.141291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572819.589487:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572820.165307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572820.613377:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572821.189322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572821.637387:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788572821.701469:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:1.0:1788572821.701528:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788572821.701545:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788572821.701583:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788572821.701595:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788572821.701606:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788572821.701616:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:0.0:1788572821.701637:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572818/real 1788572821] req@ffff94d408c54380 x1875450488569984/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572821.701666:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488569984/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572821.701687:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572821.701693:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572821.701698:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488569984:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572821.701707:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572818/real 1788572821] req@ffff94d408c57100 x1875450488569600/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572821.701720:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488569600/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572821.701727:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572821.701731:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572821.701734:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488569600:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572821.701741:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572818/real 1788572821] req@ffff94d408c56d80 x1875450488569728/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572821.701753:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488569728/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572821.701761:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572821.701765:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572821.701768:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488569728:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572821.701776:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572818/real 1788572821] req@ffff94d408c57800 x1875450488569856/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572821.701788:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488569856/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572821.701796:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572821.701800:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572821.701803:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488569856:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572821.701811:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572818/real 1788572821] req@ffff94d408c55c00 x1875450488569472/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572821.701823:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488569472/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572821.701832:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572821.701836:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572821.701839:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488569472:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788572822.213304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572822.661479:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572823.237322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572823.685383:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572823.749276:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572823.749284:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572823.749288:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572823.749304:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3972 00000100:00080000:3.0:1788572823.749313:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572823.749321:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788572823.749350:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488570112/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572823.749371:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572823.749374:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572823.749375:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572823.749378:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3972 00000100:00080000:3.0:1788572823.749381:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572823.749383:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572823.749390:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572823.749398:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488570240/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572823.749407:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572823.749411:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572823.749413:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788572823.749418:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488570112:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788572823.749420:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3972 00000100:00080000:3.0:1788572823.749424:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572823.749432:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572823.749438:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572823.749454:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488570368/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572823.749464:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572823.749468:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572823.749471:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572823.749475:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3972 00000100:00080000:3.0:1788572823.749479:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572823.749483:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572823.749490:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572823.749504:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488570496/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788572823.749505:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788572823.749518:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572823.749522:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572823.749524:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572823.749527:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3972 00000100:00080000:3.0:1788572823.749534:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572823.749538:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572823.749557:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572823.749571:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572823/real 1788572823] req@ffff94d408c55880 x1875450488570112/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572823.749573:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488570624/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572823.749632:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488570112/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572823.749644:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572823.749649:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572823.749654:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488570112:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788572823.749663:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488570240:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572823.749697:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572823.749712:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572823/real 1788572823] req@ffff94d408c56a00 x1875450488570240/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572823.749726:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488570240/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572823.749735:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572823.749739:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572823.749742:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488570240:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572823.749771:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488570368:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572823.749800:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572823.749814:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572823/real 1788572823] req@ffff94d408c55180 x1875450488570368/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572823.749827:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488570368/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572823.749835:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572823.749839:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572823.749842:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488570368:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572823.749856:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488570496:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572823.749879:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572823.749893:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572823/real 1788572823] req@ffff94d408c54000 x1875450488570496/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572823.749916:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488570496/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572823.749924:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572823.749928:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572823.749931:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488570496:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572823.749937:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488570624:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572823.749956:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572823.749969:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572823/real 1788572823] req@ffff94d408c57480 x1875450488570624/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572823.749982:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488570624/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572823.749990:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572823.749993:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572823.749996:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488570624:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572824.261312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572824.709399:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572825.285283:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572825.733411:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572826.309313:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572826.757327:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572827.333435:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572827.781404:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572828.357379:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572828.805431:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572828.869392:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572828.869400:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572828.869404:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572828.869412:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3977 00000100:00080000:3.0:1788572828.869421:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572828.869428:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788572828.869459:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488570752/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572828.869525:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488570752:192.168.202.125@tcp:250:kworker.0 00000800:00000100:0.0:1788572828.869583:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788572828.869611:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572828.869614:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572828.869616:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572828.869619:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3977 00000100:00080000:3.0:1788572828.869622:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572828.869625:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572828.869633:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572828.869660:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488570880/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572828.869704:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572828/real 1788572828] req@ffff94d408c56300 x1875450488570752/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572828.869723:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488570752/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572828.869741:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572828.869745:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572828.869750:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488570752:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788572828.869774:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488570880:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572828.869800:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572828.869814:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572828/real 1788572828] req@ffff94d408c55f80 x1875450488570880/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572828.869828:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488570880/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572828.869836:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572828.869840:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572828.869843:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488570880:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788572828.869896:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572828.869901:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572828.869908:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572828.869914:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3977 00000100:00080000:3.0:1788572828.869918:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572828.869923:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572828.869930:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572828.869945:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488571008/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572828.869962:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572828.869967:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572828.869970:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572828.869974:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3977 00000100:00080000:3.0:1788572828.869979:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788572828.869985:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488571008:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788572828.869985:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572828.869995:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572828.870010:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488571136/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572828.870021:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572828.870027:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1788572828.870028:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788572828.870031:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572828.870036:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3977 00000100:00080000:3.0:1788572828.870042:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572828.870047:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572828.870053:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572828.870060:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572828/real 1788572828] req@ffff94d408c54700 x1875450488571008/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572828.870077:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488571264/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572828.870081:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488571008/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572828.870104:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572828.870108:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572828.870112:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488571008:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572828.870128:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488571136:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572828.870156:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572828.870168:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572828/real 1788572828] req@ffff94d408c55f80 x1875450488571136/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572828.870180:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488571136/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572828.870187:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572828.870191:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572828.870194:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488571136:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572828.870200:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488571264:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572828.870241:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572828.870254:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572828/real 1788572828] req@ffff94d408c56300 x1875450488571264/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572828.870268:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488571264/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572828.870276:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572828.870280:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572828.870284:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488571264:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572829.381429:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572829.829414:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572830.405549:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572830.853334:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572831.429323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572831.877368:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572832.453581:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572832.901385:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572833.477304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572833.925324:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572833.989275:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572833.989286:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572833.989291:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572833.989300:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3982 00000100:00080000:3.0:1788572833.989309:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572833.989316:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788572833.989382:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488571392/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572833.989411:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572833.989414:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572833.989415:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572833.989419:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3982 00000100:00080000:3.0:1788572833.989422:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572833.989425:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572833.989445:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572833.989456:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488571520/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572833.989465:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488571392:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788572833.989466:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572833.989474:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572833.989476:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572833.989479:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3982 00000100:00080000:3.0:1788572833.989481:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572833.989484:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572833.989487:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572833.989495:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488571648/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572833.989517:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572833.989519:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572833.989521:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572833.989523:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3982 00000100:00080000:3.0:1788572833.989532:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572833.989534:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:0.0:1788572833.989538:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:3.0:1788572833.989538:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572833.989549:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488571776/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572833.989557:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572833.989560:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572833.989573:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572833.989578:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3982 00000100:00080000:3.0:1788572833.989582:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788572833.989589:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572833/real 1788572833] req@ffff94d408c57480 x1875450488571392/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572833.989590:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572833.989598:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572833.989617:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488571904/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572833.989621:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488571392/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572833.989649:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572833.989653:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572833.989656:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488571392:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788572833.989676:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488571520:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572833.989701:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572833.989725:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572833/real 1788572833] req@ffff94d408c54000 x1875450488571520/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572833.989739:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488571520/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572833.989757:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572833.989760:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572833.989762:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488571520:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572833.989767:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488571648:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572833.989783:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572833.989791:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572833/real 1788572833] req@ffff94d408c55180 x1875450488571648/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572833.989799:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488571648/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572833.989810:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572833.989812:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572833.989814:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488571648:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572833.989818:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488571776:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572833.989829:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572833.989835:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572833/real 1788572833] req@ffff94d408c56a00 x1875450488571776/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572833.989842:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488571776/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572833.989847:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572833.989849:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572833.989851:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488571776:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572833.989854:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488571904:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572833.989863:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572833.989869:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572833/real 1788572833] req@ffff94d408c55880 x1875450488571904/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572833.989876:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488571904/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572833.989881:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572833.989882:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572833.989885:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488571904:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572834.501274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572834.949327:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572835.525322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572835.973446:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572836.549287:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572836.997366:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572837.573298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572838.021414:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572838.597281:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572839.045433:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572839.109439:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572839.109451:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572839.109455:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572839.109467:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3987 00000100:00080000:3.0:1788572839.109481:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572839.109490:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788572839.109546:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488572032/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572839.109584:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572839.109589:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572839.109591:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572839.109596:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3987 00000100:00080000:3.0:1788572839.109600:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572839.109606:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572839.109618:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572839.109633:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488572160/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572839.109644:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572839.109648:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572839.109650:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572839.109655:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3987 00000100:00080000:3.0:1788572839.109658:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572839.109663:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572839.109667:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572839.109694:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488572032:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788572839.109705:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488572288/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572839.109721:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572839.109726:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572839.109729:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572839.109734:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3987 00000100:00080000:3.0:1788572839.109755:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572839.109760:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572839.109766:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572839.109780:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488572416/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572839.109790:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572839.109793:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572839.109794:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572839.109798:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3987 00000100:00080000:3.0:1788572839.109801:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788572839.109832:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788572839.109833:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572839.109873:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572839.109915:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488572544/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572839.109945:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572839/real 1788572839] req@ffff94d408c55c00 x1875450488572032/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572839.109965:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488572032/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572839.109975:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572839.109980:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572839.109983:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488572032:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788572839.109988:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488572160:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572839.110006:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572839.110015:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572839/real 1788572839] req@ffff94d408c57800 x1875450488572160/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572839.110023:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488572160/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572839.110028:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572839.110030:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572839.110032:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488572160:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572839.110057:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488572288:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572839.110068:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572839.110075:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572839/real 1788572839] req@ffff94d408c56d80 x1875450488572288/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572839.110083:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488572288/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572839.110089:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572839.110090:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572839.110092:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488572288:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572839.110096:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488572416:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572839.110108:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572839.110113:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572839/real 1788572839] req@ffff94d408c57100 x1875450488572416/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572839.110121:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488572416/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572839.110126:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572839.110128:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572839.110130:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488572416:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572839.110133:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488572544:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572839.110156:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572839.110163:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572839/real 1788572839] req@ffff94d408c54380 x1875450488572544/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572839.110170:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488572544/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572839.110177:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572839.110179:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572839.110181:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488572544:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572839.621309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572840.069379:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572840.645265:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572841.093370:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572841.669252:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572842.117369:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572842.693301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572843.141370:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572843.717283:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572844.165351:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572844.229304:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572844.229313:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572844.229317:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572844.229325:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3992 00000100:00080000:3.0:1788572844.229334:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572844.229341:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788572844.229373:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488572672/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572844.229398:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572844.229401:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572844.229402:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572844.229404:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3992 00000100:00080000:3.0:1788572844.229407:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572844.229413:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572844.229420:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572844.229428:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488572800/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572844.229436:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572844.229439:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572844.229440:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572844.229443:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3992 00000100:00080000:3.0:1788572844.229446:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572844.229448:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572844.229452:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572844.229455:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488572672:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788572844.229462:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488572928/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572844.229470:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572844.229472:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572844.229474:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572844.229476:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3992 00000100:00080000:3.0:1788572844.229478:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572844.229480:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572844.229483:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572844.229491:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55500 x1875450488573056/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572844.229497:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572844.229500:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572844.229501:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572844.229504:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3992 00000100:00080000:3.0:1788572844.229506:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572844.229508:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572844.229511:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788572844.229515:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572844.229521:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488573184/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572844.229549:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572844/real 1788572844] req@ffff94d408c54e00 x1875450488572672/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572844.229564:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488572672/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572844.229573:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572844.229575:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572844.229578:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488572672:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788572844.229583:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488572800:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572844.229598:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572844.229606:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572844/real 1788572844] req@ffff94d408c57b80 x1875450488572800/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572844.229614:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488572800/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572844.229620:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572844.229622:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572844.229624:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488572800:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572844.229642:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488572928:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572844.229653:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572844.229659:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572844/real 1788572844] req@ffff94d408c54a80 x1875450488572928/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572844.229667:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488572928/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572844.229672:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572844.229675:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572844.229677:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488572928:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572844.229680:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488573056:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572844.229690:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572844.229695:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572844/real 1788572844] req@ffff94d408c55500 x1875450488573056/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572844.229702:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55500 x1875450488573056/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572844.229708:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572844.229709:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572844.229711:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488573056:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572844.229715:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488573184:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572844.229724:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572844.229730:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572844/real 1788572844] req@ffff94d408c56680 x1875450488573184/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572844.229738:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488573184/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572844.229743:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572844.229744:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572844.229746:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488573184:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572844.741269:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572845.189354:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572845.765289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572846.213383:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572846.789277:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572847.237411:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572847.813320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572848.261398:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572848.837284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572849.285428:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572849.349283:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572849.349292:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572849.349296:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572849.349304:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 3997 00000100:00080000:3.0:1788572849.349313:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572849.349320:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788572849.349351:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488573312/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572849.349374:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572849.349377:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572849.349379:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572849.349382:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3997 00000100:00080000:3.0:1788572849.349385:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572849.349388:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572849.349396:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572849.349406:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55500 x1875450488573440/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572849.349414:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488573312:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788572849.349429:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572849.349433:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572849.349434:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572849.349439:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3997 00000100:00080000:3.0:1788572849.349442:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572849.349446:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572849.349450:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572849.349464:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488573568/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572849.349472:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572849.349475:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572849.349477:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788572849.349485:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788572849.349486:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3997 00000100:00080000:3.0:1788572849.349492:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572849.349495:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572849.349499:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572849.349509:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488573696/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572849.349523:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572849/real 1788572849] req@ffff94d408c56680 x1875450488573312/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572849.349538:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488573312/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572849.349547:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572849.349550:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572849.349554:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488573312:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788572849.349581:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572849.349585:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572849.349587:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788572849.349589:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488573440:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788572849.349594:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 3997 00000100:00080000:3.0:1788572849.349599:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572849.349603:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572849.349609:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788572849.349616:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572849.349624:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488573824/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572849.349629:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572849/real 1788572849] req@ffff94d408c55500 x1875450488573440/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572849.349638:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55500 x1875450488573440/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572849.349647:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572849.349649:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572849.349652:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488573440:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572849.349664:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488573568:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572849.349680:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572849.349688:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572849/real 1788572849] req@ffff94d408c54a80 x1875450488573568/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572849.349696:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488573568/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572849.349701:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572849.349703:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572849.349705:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488573568:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572849.349709:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488573696:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572849.349720:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572849.349726:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572849/real 1788572849] req@ffff94d408c57b80 x1875450488573696/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572849.349733:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488573696/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572849.349737:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572849.349739:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572849.349742:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488573696:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572849.349754:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488573824:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572849.349768:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572849.349775:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572849/real 1788572849] req@ffff94d408c54e00 x1875450488573824/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572849.349783:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488573824/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572849.349789:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572849.349792:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572849.349794:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488573824:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572849.861321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572850.309390:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572850.885275:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572851.333349:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572851.909273:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572852.357394:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572852.933289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572853.381365:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572853.957294:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572854.405338:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572854.469304:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572854.469313:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572854.469316:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572854.469324:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4002 00000100:00080000:2.0:1788572854.469333:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572854.469339:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788572854.469371:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488573952/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572854.469395:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572854.469398:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572854.469399:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572854.469402:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4002 00000100:00080000:2.0:1788572854.469404:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572854.469406:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572854.469413:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572854.469421:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488574080/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572854.469431:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572854.469436:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788572854.469440:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488573952:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788572854.469443:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572854.469448:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4002 00000100:00080000:2.0:1788572854.469450:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572854.469454:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572854.469458:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572854.469466:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488574208/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572854.469472:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572854.469475:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572854.469476:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572854.469478:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4002 00000100:00080000:2.0:1788572854.469480:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572854.469482:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572854.469485:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572854.469498:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63100 x1875450488574336/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572854.469504:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572854.469507:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572854.469508:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572854.469510:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4002 00000100:00080000:2.0:1788572854.469512:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572854.469514:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:0.0:1788572854.469520:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:2.0:1788572854.469521:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572854.469531:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60e00 x1875450488574464/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572854.469556:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572854/real 1788572854] req@ffff94d408c61f80 x1875450488573952/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572854.469571:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488573952/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572854.469579:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572854.469582:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572854.469585:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488573952:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788572854.469603:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488574080:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572854.469620:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572854.469629:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572854/real 1788572854] req@ffff94d408c61880 x1875450488574080/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572854.469637:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488574080/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572854.469643:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572854.469645:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572854.469647:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488574080:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572854.469651:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488574208:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572854.469661:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572854.469667:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572854/real 1788572854] req@ffff94d408c63480 x1875450488574208/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572854.469674:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488574208/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572854.469680:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572854.469681:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572854.469683:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488574208:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572854.469687:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488574336:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572854.469697:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572854.469703:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572854/real 1788572854] req@ffff94d408c63100 x1875450488574336/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572854.469710:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63100 x1875450488574336/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572854.469715:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572854.469717:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572854.469719:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488574336:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572854.469723:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488574464:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572854.469732:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572854.469737:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572854/real 1788572854] req@ffff94d408c60e00 x1875450488574464/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572854.469744:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60e00 x1875450488574464/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572854.469749:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572854.469751:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572854.469754:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488574464:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572854.981323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572855.429383:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572856.005262:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572856.453339:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572857.029299:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572857.477399:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572858.053301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572858.501370:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572859.077318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572859.525397:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572859.589303:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572859.589313:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572859.589326:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572859.589336:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4007 00000100:00080000:3.0:1788572859.589347:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572859.589355:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788572859.589388:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488574592/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572859.589410:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572859.589413:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572859.589414:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572859.589417:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4007 00000100:00080000:3.0:1788572859.589420:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572859.589422:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572859.589429:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572859.589439:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488574720/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572859.589450:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488574592:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788572859.589455:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572859.589458:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572859.589460:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572859.589463:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4007 00000100:00080000:3.0:1788572859.589465:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572859.589467:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572859.589471:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572859.589480:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488574848/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572859.589487:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572859.589490:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572859.589491:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572859.589494:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4007 00000100:00080000:3.0:1788572859.589503:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572859.589505:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572859.589509:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788572859.589514:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572859.589520:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488574976/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572859.589527:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572859.589530:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572859.589531:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572859.589533:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4007 00000100:00080000:3.0:1788572859.589538:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572859.589540:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572859.589543:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572859.589550:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488575104/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572859.589559:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572859/real 1788572859] req@ffff94d408c54380 x1875450488574592/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572859.589578:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488574592/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572859.589586:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572859.589589:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572859.589592:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488574592:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788572859.589619:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488574720:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572859.589638:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572859.589647:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572859/real 1788572859] req@ffff94d408c57100 x1875450488574720/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572859.589656:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488574720/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572859.589662:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572859.589664:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572859.589666:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488574720:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572859.589670:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488574848:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572859.589680:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572859.589687:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572859/real 1788572859] req@ffff94d408c56d80 x1875450488574848/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572859.589694:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488574848/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572859.589699:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572859.589701:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572859.589703:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488574848:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572859.589707:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488574976:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572859.589726:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572859.589734:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572859/real 1788572859] req@ffff94d408c57800 x1875450488574976/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572859.589742:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488574976/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572859.589753:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572859.589755:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572859.589757:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488574976:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572859.589761:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488575104:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572859.589772:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572859.589778:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572859/real 1788572859] req@ffff94d408c55c00 x1875450488575104/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572859.589786:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488575104/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572859.589791:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572859.589794:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572859.589796:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488575104:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572860.101355:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572860.549418:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572861.125367:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572861.573369:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572862.149357:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572862.597386:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572863.173367:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572863.621453:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572864.197306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572864.645397:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572864.709315:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572864.709329:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572864.709335:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572864.709348:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4013 00000100:00080000:3.0:1788572864.709376:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572864.709387:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788572864.709437:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488575232/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572864.709473:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572864.709478:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572864.709480:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572864.709484:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4013 00000100:00080000:3.0:1788572864.709488:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572864.709493:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572864.709505:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572864.709522:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488575360/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572864.709526:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488575232:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788572864.709544:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572864.709549:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572864.709551:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572864.709557:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4013 00000100:00080000:3.0:1788572864.709561:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572864.709566:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572864.709572:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572864.709588:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488575488/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572864.709600:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572864.709605:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572864.709607:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788572864.709610:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788572864.709615:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4013 00000100:00080000:3.0:1788572864.709621:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572864.709626:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572864.709633:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572864.709647:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488575616/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572864.709656:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572864/real 1788572864] req@ffff94d408c55880 x1875450488575232/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572864.709660:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572864.709670:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572864.709673:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572864.709680:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4013 00000100:00100000:0.0:1788572864.709682:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488575232/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572864.709686:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572864.709696:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00080000:0.0:1788572864.709703:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788572864.709706:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788572864.709715:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572864.709722:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488575232:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788572864.709722:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488575744/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572864.709753:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488575360:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572864.709772:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572864.709781:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572864/real 1788572864] req@ffff94d408c56a00 x1875450488575360/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572864.709790:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488575360/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572864.709796:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572864.709798:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572864.709800:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488575360:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572864.709804:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488575488:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572864.709815:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572864.709821:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572864/real 1788572864] req@ffff94d408c55180 x1875450488575488/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572864.709829:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488575488/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572864.709834:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572864.709836:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572864.709838:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488575488:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572864.709842:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488575616:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572864.709853:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572864.709859:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572864/real 1788572864] req@ffff94d408c54000 x1875450488575616/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572864.709866:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488575616/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572864.709871:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572864.709873:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572864.709875:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488575616:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572864.709878:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488575744:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572864.709888:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572864.709893:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572864/real 1788572864] req@ffff94d408c57480 x1875450488575744/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572864.709901:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488575744/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572864.709906:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572864.709907:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572864.709909:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488575744:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572865.221305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572865.669367:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572866.245358:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572866.693358:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572867.269272:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572867.717368:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572868.293314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572868.741348:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572869.317295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572869.765368:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572869.829433:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572869.829445:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572869.829450:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572869.829485:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4018 00000100:00080000:0.0:1788572869.829499:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572869.829510:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788572869.829566:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488575872/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572869.829603:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572869.829607:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572869.829610:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572869.829614:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4018 00000100:00080000:0.0:1788572869.829618:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572869.829623:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572869.829635:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572869.829651:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f800 x1875450488576000/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572869.829659:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488575872:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788572869.829671:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572869.829674:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572869.829676:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572869.829680:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4018 00000100:00080000:0.0:1788572869.829684:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572869.829688:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572869.829692:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572869.829704:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488576128/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572869.829716:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572869.829719:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572869.829720:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572869.829723:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4018 00000100:00080000:0.0:1788572869.829726:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572869.829729:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572869.829732:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788572869.829740:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572869.829741:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f100 x1875450488576256/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572869.829749:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572869.829752:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572869.829754:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572869.829757:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4018 00000100:00080000:0.0:1788572869.829760:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572869.829762:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572869.829765:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572869.829772:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488576384/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572869.829795:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572869/real 1788572869] req@ffff94d408d3d500 x1875450488575872/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572869.829820:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488575872/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572869.829839:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572869.829844:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572869.829850:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488575872:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788572869.829878:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488576000:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572869.829900:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572869.829910:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572869/real 1788572869] req@ffff94d408d3f800 x1875450488576000/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572869.829919:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f800 x1875450488576000/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572869.829925:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572869.829928:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572869.829930:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488576000:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572869.829935:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488576128:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572869.829946:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572869.829963:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572869/real 1788572869] req@ffff94d408d3ca80 x1875450488576128/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572869.829971:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488576128/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572869.829977:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572869.829979:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572869.829982:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488576128:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572869.829986:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488576256:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572869.830000:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572869.830006:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572869/real 1788572869] req@ffff94d408d3f100 x1875450488576256/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572869.830014:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f100 x1875450488576256/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572869.830019:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572869.830022:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572869.830025:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488576256:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572869.830028:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488576384:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572869.830038:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572869.830043:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572869/real 1788572869] req@ffff94d408d3e680 x1875450488576384/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572869.830050:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488576384/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572869.830055:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572869.830058:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572869.830060:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488576384:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572870.341263:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572870.789441:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572871.365328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572871.813359:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572872.389300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572872.837362:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572873.413292:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572873.861324:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572874.437322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572874.885377:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572874.949315:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572874.949324:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572874.949328:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572874.949336:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4023 00000100:00080000:3.0:1788572874.949345:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572874.949351:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788572874.949383:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488576512/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572874.949405:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572874.949408:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572874.949410:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572874.949412:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4023 00000100:00080000:3.0:1788572874.949415:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572874.949417:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572874.949424:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572874.949433:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488576640/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572874.949441:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572874.949443:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572874.949447:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788572874.949448:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488576512:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788572874.949452:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4023 00000100:00080000:3.0:1788572874.949454:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572874.949457:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572874.949460:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572874.949468:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488576768/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572874.949475:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572874.949478:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572874.949479:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572874.949482:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4023 00000100:00080000:3.0:1788572874.949485:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572874.949487:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572874.949491:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572874.949501:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488576896/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572874.949510:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572874.949513:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572874.949514:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788572874.949522:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788572874.949526:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4023 00000100:00080000:3.0:1788572874.949534:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572874.949539:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572874.949545:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572874.949573:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488577024/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572874.949583:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572874/real 1788572874] req@ffff94d408c56300 x1875450488576512/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572874.949605:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488576512/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572874.949617:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572874.949621:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572874.949631:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488576512:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788572874.949655:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488576640:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572874.949680:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572874.949704:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572874/real 1788572874] req@ffff94d408c55f80 x1875450488576640/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572874.949718:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488576640/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572874.949727:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572874.949731:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572874.949734:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488576640:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572874.949741:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488576768:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572874.949775:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572874.949789:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572874/real 1788572874] req@ffff94d408c54700 x1875450488576768/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572874.949802:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488576768/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572874.949810:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572874.949814:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572874.949817:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488576768:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572874.949824:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488576896:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572874.949845:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572874.949856:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572874/real 1788572874] req@ffff94d408c57480 x1875450488576896/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572874.949868:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488576896/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572874.949876:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572874.949879:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572874.949883:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488576896:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572874.949889:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488577024:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572874.949909:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572874.949920:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572874/real 1788572874] req@ffff94d408c54000 x1875450488577024/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572874.949932:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488577024/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572874.949947:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572874.949950:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572874.949953:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488577024:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572875.461290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572875.909361:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572876.485316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572876.933342:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572877.509286:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572877.957363:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572878.533301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572878.981369:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572879.557321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572880.005391:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572880.069418:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572880.069429:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572880.069434:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572880.069445:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4028 00000100:00080000:2.0:1788572880.069458:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572880.069466:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788572880.069521:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488577152/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572880.069551:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572880.069554:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572880.069555:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572880.069558:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4028 00000100:00080000:2.0:1788572880.069562:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572880.069565:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572880.069573:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572880.069585:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488577280/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572880.069593:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572880.069596:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572880.069598:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572880.069604:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4028 00000100:00080000:2.0:1788572880.069609:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572880.069612:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572880.069615:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572880.069623:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488577408/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572880.069654:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488577152:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788572880.069654:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572880.069664:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572880.069669:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572880.069674:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4028 00000100:00080000:2.0:1788572880.069677:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572880.069682:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572880.069688:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572880.069702:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488577536/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572880.069710:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572880.069714:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572880.069715:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572880.069718:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4028 00000100:00080000:2.0:1788572880.069720:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572880.069723:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572880.069727:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788572880.069736:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788572880.069737:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488577664/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572880.069783:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572880/real 1788572880] req@ffff94d408c61c00 x1875450488577152/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572880.069798:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488577152/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572880.069807:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572880.069810:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572880.069814:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488577152:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788572880.069820:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488577280:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572880.069836:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572880.069844:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572880/real 1788572880] req@ffff94d408c61500 x1875450488577280/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572880.069852:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488577280/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572880.069858:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572880.069860:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572880.069862:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488577280:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572880.069881:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488577408:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572880.069892:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572880.069898:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572880/real 1788572880] req@ffff94d408c63b80 x1875450488577408/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572880.069906:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488577408/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572880.069911:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572880.069913:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572880.069915:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488577408:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572880.069919:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488577536:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572880.069929:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572880.069935:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572880/real 1788572880] req@ffff94d408c62a00 x1875450488577536/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572880.069942:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488577536/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572880.069947:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572880.069949:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572880.069951:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488577536:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572880.069955:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488577664:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572880.069965:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572880.069970:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572880/real 1788572880] req@ffff94d408c63800 x1875450488577664/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572880.069977:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488577664/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572880.069982:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572880.069984:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572880.069986:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488577664:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572880.581355:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572881.029419:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572881.605357:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572882.053496:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572882.629306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572883.077369:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572883.653323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572884.101387:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572884.677324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572885.125416:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572885.189352:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572885.189367:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572885.189373:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572885.189386:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4033 00000100:00080000:2.0:1788572885.189407:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572885.189420:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788572885.189465:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488577792/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572885.189497:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572885.189501:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572885.189503:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572885.189507:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4033 00000100:00080000:2.0:1788572885.189511:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572885.189516:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572885.189535:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572885.189551:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488577792:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788572885.189552:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488577920/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572885.189569:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572885.189573:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572885.189575:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572885.189579:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4033 00000100:00080000:2.0:1788572885.189583:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572885.189588:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572885.189595:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572885.189609:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488578048/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572885.189618:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572885.189621:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572885.189623:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572885.189627:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4033 00000100:00080000:2.0:1788572885.189633:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572885.189637:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572885.189644:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572885.189655:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488578176/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572885.189667:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788572885.189671:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488577920:192.168.202.125@tcp:38:kworker.0 00000100:00080000:2.0:1788572885.189673:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572885.189677:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572885.189684:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4033 00000100:00080000:2.0:1788572885.189701:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572885.189707:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572885.189713:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572885.189718:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488578048:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788572885.189727:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488578304/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572885.189739:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488578176:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572885.189770:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488578304:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572885.701348:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572886.149410:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572886.725352:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572887.173424:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572887.749286:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572888.197439:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788572888.261479:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:3.0:1788572888.261526:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788572888.261542:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:3.0:1788572888.261596:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:3.0:1788572888.261607:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:3.0:1788572888.261615:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:3.0:1788572888.261623:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:0.0:1788572888.261755:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572885/real 1788572888] req@ffff94d408c62680 x1875450488578304/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572888.261788:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488578304/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572888.261804:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572888.261810:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572888.261816:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488578304:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572888.261827:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572885/real 1788572888] req@ffff94d408c61180 x1875450488578176/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572888.261839:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488578176/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572888.261847:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572888.261851:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572888.261854:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488578176:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572888.261861:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572885/real 1788572888] req@ffff94d408c62d80 x1875450488577920/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572888.261874:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488577920/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572888.261889:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572888.261893:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572888.261897:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488577920:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572888.261906:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572885/real 1788572888] req@ffff94d408c60000 x1875450488578048/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572888.261918:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488578048/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572888.261926:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572888.261930:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572888.261932:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488578048:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572888.261938:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572885/real 1788572888] req@ffff94d408c60a80 x1875450488577792/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572888.261947:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488577792/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572888.261952:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572888.261955:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572888.261958:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488577792:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788572888.773314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572889.221353:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572889.797281:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572890.245348:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572890.309346:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572890.309356:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572890.309360:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572890.309374:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4038 00000100:00080000:1.0:1788572890.309388:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572890.309397:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788572890.309445:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488578432/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572890.309479:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572890.309483:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572890.309485:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572890.309490:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4038 00000100:00080000:1.0:1788572890.309494:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572890.309498:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572890.309510:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572890.309524:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488578560/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572890.309536:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572890.309542:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788572890.309545:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488578432:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788572890.309545:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572890.309559:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4038 00000100:00080000:1.0:1788572890.309563:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572890.309568:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572890.309573:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572890.309602:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488578688/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572890.309614:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572890.309618:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572890.309621:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572890.309627:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4038 00000100:00080000:1.0:1788572890.309631:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572890.309643:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572890.309650:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788572890.309657:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572890.309665:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488578816/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572890.309677:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572890.309681:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572890.309684:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572890.309689:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4038 00000100:00080000:1.0:1788572890.309693:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572890.309698:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572890.309705:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572890.309719:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488578944/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572890.309728:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572890/real 1788572890] req@ffff94d4049be680 x1875450488578432/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572890.309770:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488578432/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572890.309784:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572890.309795:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572890.309800:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488578432:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788572890.309835:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488578560:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572890.309861:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572890.309874:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572890/real 1788572890] req@ffff94d4049bca80 x1875450488578560/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572890.309898:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488578560/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572890.309906:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572890.309909:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572890.309912:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488578560:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572890.309917:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488578688:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572890.309935:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572890.309944:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572890/real 1788572890] req@ffff94d4049bc700 x1875450488578688/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572890.309954:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488578688/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572890.309968:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572890.309972:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572890.309974:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488578688:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572890.309980:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488578816:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572890.310001:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572890.310011:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572890/real 1788572890] req@ffff94d4049bf100 x1875450488578816/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572890.310022:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488578816/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572890.310031:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572890.310034:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572890.310043:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488578816:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572890.310051:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488578944:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572890.310071:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572890.310083:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572890/real 1788572890] req@ffff94d4049bd500 x1875450488578944/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572890.310095:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488578944/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572890.310103:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572890.310106:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572890.310110:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488578944:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572890.821297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572891.269383:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572891.845313:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572892.293395:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572892.869345:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572893.317324:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572893.893309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572894.341422:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572894.917337:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572895.365339:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572895.429362:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572895.429373:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572895.429379:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572895.429390:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4043 00000100:00080000:2.0:1788572895.429402:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572895.429409:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788572895.429458:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60700 x1875450488579072/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572895.429482:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572895.429485:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572895.429486:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572895.429489:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4043 00000100:00080000:2.0:1788572895.429491:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572895.429493:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572895.429501:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572895.429510:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60380 x1875450488579200/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572895.429517:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572895.429520:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572895.429521:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572895.429526:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4043 00000100:00100000:0.0:1788572895.429528:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488579072:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788572895.429528:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572895.429534:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572895.429539:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572895.429546:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62300 x1875450488579328/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572895.429556:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572895.429559:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572895.429560:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572895.429562:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4043 00000100:00080000:2.0:1788572895.429564:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572895.429566:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572895.429569:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572895.429577:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488579456/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572895.429582:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572895.429584:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572895.429585:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572895.429588:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4043 00000100:00080000:2.0:1788572895.429592:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572895.429594:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572895.429597:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572895.429604:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488579584/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788572895.429609:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572895.429652:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572895/real 1788572895] req@ffff94d408c60700 x1875450488579072/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572895.429666:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60700 x1875450488579072/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572895.429674:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572895.429677:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572895.429680:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488579072:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788572895.429701:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488579200:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572895.429717:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572895.429724:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572895/real 1788572895] req@ffff94d408c60380 x1875450488579200/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572895.429732:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60380 x1875450488579200/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572895.429737:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572895.429739:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572895.429741:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488579200:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572895.429744:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488579328:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572895.429753:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572895.429759:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572895/real 1788572895] req@ffff94d408c62300 x1875450488579328/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572895.429771:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62300 x1875450488579328/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572895.429776:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572895.429778:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572895.429780:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488579328:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572895.429783:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488579456:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572895.429793:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572895.429798:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572895/real 1788572895] req@ffff94d408c60a80 x1875450488579456/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572895.429824:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488579456/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572895.429829:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572895.429831:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572895.429833:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488579456:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572895.429837:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488579584:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572895.429850:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572895.429856:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572895/real 1788572895] req@ffff94d408c60000 x1875450488579584/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572895.429863:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488579584/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572895.429868:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572895.429870:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572895.429872:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488579584:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572895.941565:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572896.389385:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572896.966182:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572897.413369:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572897.989447:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572898.437338:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572899.013330:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572899.461372:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572900.037305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572900.485420:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572900.549302:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572900.549314:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572900.549318:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572900.549329:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4048 00000100:00080000:3.0:1788572900.549341:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572900.549349:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788572900.549389:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488579712/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572900.549417:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572900.549422:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572900.549424:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572900.549428:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4048 00000100:00080000:3.0:1788572900.549433:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572900.549438:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572900.549448:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572900.549458:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488579712:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788572900.549465:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488579840/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572900.549480:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572900.549483:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572900.549486:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572900.549491:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4048 00000100:00080000:3.0:1788572900.549496:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572900.549500:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572900.549506:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572900.549521:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488579968/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788572900.549525:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788572900.549537:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572900.549541:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572900.549543:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572900.549548:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4048 00000100:00080000:3.0:1788572900.549553:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572900.549558:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572900.549564:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572900.549571:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572900/real 1788572900] req@ffff94d408c55180 x1875450488579712/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572900.549578:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488580096/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572900.549600:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788572900.549603:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488579712/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572900.549604:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572900.549607:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572900.549610:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4048 00000100:00080000:0.0:1788572900.549622:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572900.549623:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572900.549630:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00080000:0.0:1788572900.549631:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572900.549639:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488579712:192.168.202.125@tcp:250:kworker.0 00000008:00100000:3.0:1788572900.549639:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572900.549654:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488580224/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572900.549667:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488579840:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572900.549688:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572900.549702:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572900/real 1788572900] req@ffff94d408c56a00 x1875450488579840/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572900.549715:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488579840/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572900.549724:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572900.549728:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572900.549732:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488579840:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572900.549740:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488579968:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572900.549759:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572900.549767:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572900/real 1788572900] req@ffff94d408c55880 x1875450488579968/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572900.549776:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488579968/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572900.549782:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572900.549784:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572900.549786:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488579968:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572900.549790:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488580096:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572900.549801:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572900.549806:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572900/real 1788572900] req@ffff94d408c55c00 x1875450488580096/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572900.549814:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488580096/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572900.549819:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572900.549820:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572900.549822:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488580096:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572900.549826:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488580224:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572900.549835:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572900.549840:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572900/real 1788572900] req@ffff94d408c57800 x1875450488580224/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572900.549848:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488580224/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572900.549853:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572900.549854:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572900.549874:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488580224:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572901.061302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572901.509351:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572902.085293:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572902.533355:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572903.109273:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572903.557361:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572904.133291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572904.581370:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572905.157310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572905.605380:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572905.669300:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572905.669311:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572905.669316:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572905.669328:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4053 00000100:00080000:3.0:1788572905.669339:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572905.669349:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788572905.669392:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488580352/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572905.669424:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572905.669429:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572905.669432:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572905.669436:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4053 00000100:00080000:3.0:1788572905.669441:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572905.669445:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572905.669456:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572905.669472:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488580480/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572905.669486:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788572905.669492:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488580352:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788572905.669492:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572905.669496:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572905.669505:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4053 00000100:00080000:3.0:1788572905.669509:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572905.669514:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572905.669520:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572905.669537:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488580608/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572905.669548:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572905.669552:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572905.669555:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572905.669560:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4053 00000100:00080000:3.0:1788572905.669564:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572905.669569:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:0.0:1788572905.669574:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:3.0:1788572905.669578:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572905.669595:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488580736/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572905.669607:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572905.669610:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572905.669613:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572905.669617:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4053 00000100:00100000:0.0:1788572905.669622:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572905/real 1788572905] req@ffff94d408c56d80 x1875450488580352/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572905.669624:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572905.669633:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788572905.669651:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488580352/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788572905.669651:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788572905.669667:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788572905.669671:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488580864/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572905.669675:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572905.669683:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488580352:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788572905.669717:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488580480:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572905.669744:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572905.669757:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572905/real 1788572905] req@ffff94d408c57100 x1875450488580480/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572905.669771:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488580480/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572905.669786:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572905.669789:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572905.669793:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488580480:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572905.669800:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488580608:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572905.669819:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572905.669831:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572905/real 1788572905] req@ffff94d408c54380 x1875450488580608/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572905.669843:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488580608/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572905.669852:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572905.669855:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572905.669859:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488580608:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572905.669865:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488580736:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572905.669886:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572905.669899:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572905/real 1788572905] req@ffff94d408c54e00 x1875450488580736/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572905.669940:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488580736/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572905.669950:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572905.669953:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572905.669958:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488580736:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572905.669965:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488580864:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572905.669987:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572905.670000:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572905/real 1788572905] req@ffff94d408c57b80 x1875450488580864/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572905.670013:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488580864/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572905.670021:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572905.670024:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572905.670027:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488580864:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572906.181304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572906.629381:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572907.205263:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572907.653368:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572908.229273:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572908.677346:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572909.253281:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572909.701394:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572910.277269:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572910.725318:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572910.789310:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572910.789323:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572910.789326:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572910.789334:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4059 00000100:00080000:3.0:1788572910.789343:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572910.789349:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788572910.789383:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488580992/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572910.789410:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572910.789414:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572910.789415:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572910.789417:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4059 00000100:00080000:3.0:1788572910.789420:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572910.789422:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572910.789429:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572910.789438:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55500 x1875450488581120/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572910.789446:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572910.789448:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572910.789450:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572910.789452:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4059 00000100:00080000:3.0:1788572910.789454:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572910.789458:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788572910.789461:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488580992:192.168.202.125@tcp:250:kworker.0 00000008:00100000:3.0:1788572910.789462:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572910.789473:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488581248/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572910.789481:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572910.789484:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572910.789485:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572910.789487:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4059 00000100:00080000:3.0:1788572910.789490:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572910.789493:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572910.789497:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572910.789505:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488581376/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572910.789511:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572910.789513:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572910.789514:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572910.789516:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4059 00000100:00080000:3.0:1788572910.789519:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788572910.789523:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788572910.789525:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572910.789530:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572910.789536:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488581504/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572910.789569:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572910/real 1788572910] req@ffff94d408c54a80 x1875450488580992/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572910.789584:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488580992/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572910.789592:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572910.789595:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572910.789598:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488580992:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788572910.789603:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488581120:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572910.789619:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572910.789628:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572910/real 1788572910] req@ffff94d408c55500 x1875450488581120/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572910.789636:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55500 x1875450488581120/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572910.789641:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572910.789643:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572910.789646:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488581120:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572910.789663:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488581248:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572910.789673:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572910.789680:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572910/real 1788572910] req@ffff94d408c56680 x1875450488581248/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572910.789687:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488581248/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572910.789692:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572910.789694:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572910.789696:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488581248:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572910.789700:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488581376:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572910.789710:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572910.789716:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572910/real 1788572910] req@ffff94d408c57b80 x1875450488581376/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572910.789725:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488581376/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572910.789734:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572910.789737:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572910.789740:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488581376:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572910.789746:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488581504:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572910.789767:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572910.789779:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572910/real 1788572910] req@ffff94d408c54e00 x1875450488581504/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572910.789789:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488581504/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572910.789795:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572910.789797:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572910.789799:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488581504:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572911.301259:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572911.749386:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572912.325267:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572912.773329:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572913.349332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572913.797376:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572914.373309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572914.821376:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572915.397316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572915.845433:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572915.909330:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572915.909341:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572915.909346:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572915.909357:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4064 00000100:00080000:2.0:1788572915.909370:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572915.909378:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788572915.909423:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488581632/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572915.909452:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572915.909455:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572915.909457:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572915.909460:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4064 00000100:00080000:2.0:1788572915.909462:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572915.909464:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572915.909474:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572915.909486:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488581760/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572915.909494:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572915.909497:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572915.909499:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572915.909502:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4064 00000100:00080000:2.0:1788572915.909504:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572915.909507:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572915.909512:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572915.909519:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488581888/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572915.909527:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788572915.909531:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488581632:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788572915.909532:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572915.909536:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572915.909545:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4064 00000100:00080000:2.0:1788572915.909549:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572915.909552:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572915.909555:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572915.909563:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488582016/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572915.909569:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572915.909572:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572915.909574:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572915.909576:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4064 00000100:00080000:2.0:1788572915.909579:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572915.909581:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572915.909584:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572915.909590:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488582144/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788572915.909622:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572915.909660:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572915/real 1788572915] req@ffff94d408c62d80 x1875450488581632/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572915.909679:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488581632/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572915.909690:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572915.909695:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572915.909700:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488581632:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788572915.909709:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488581760:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572915.909734:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572915.909748:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572915/real 1788572915] req@ffff94d408c61180 x1875450488581760/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572915.909762:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488581760/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572915.909771:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572915.909775:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572915.909778:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488581760:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572915.909804:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488581888:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572915.909822:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572915.909830:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572915/real 1788572915] req@ffff94d408c62680 x1875450488581888/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572915.909838:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488581888/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572915.909844:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572915.909846:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572915.909848:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488581888:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572915.909852:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488582016:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572915.909863:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572915.909869:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572915/real 1788572915] req@ffff94d408c63800 x1875450488582016/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572915.909877:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488582016/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572915.909882:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572915.909883:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572915.909885:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488582016:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572915.909889:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488582144:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572915.909898:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572915.909903:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572915/real 1788572915] req@ffff94d408c62a00 x1875450488582144/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572915.909911:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488582144/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572915.909916:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572915.909917:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572915.909919:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488582144:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572916.421274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572916.869439:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572917.445342:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572917.893471:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572918.469309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572918.917327:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572919.493297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572919.941340:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572920.517299:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572920.965365:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572921.029312:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572921.029322:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572921.029327:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572921.029337:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4069 00000100:00080000:2.0:1788572921.029348:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572921.029356:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788572921.029389:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488582272/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572921.029425:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572921.029429:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572921.029431:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572921.029435:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4069 00000100:00080000:2.0:1788572921.029439:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572921.029444:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572921.029454:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572921.029468:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488582400/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572921.029479:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572921.029482:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572921.029487:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788572921.029488:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488582272:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788572921.029493:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4069 00000100:00080000:2.0:1788572921.029501:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572921.029505:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572921.029511:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572921.029526:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488582528/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572921.029535:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572921.029539:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572921.029541:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572921.029546:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4069 00000100:00080000:2.0:1788572921.029623:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572921.029628:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572921.029634:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788572921.029636:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788572921.029651:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60e00 x1875450488582656/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572921.029663:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572921.029667:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572921.029669:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572921.029673:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4069 00000100:00080000:2.0:1788572921.029678:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788572921.029683:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572921/real 1788572921] req@ffff94d408c63b80 x1875450488582272/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572921.029685:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572921.029694:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572921.029709:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63100 x1875450488582784/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572921.029723:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488582272/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572921.029736:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572921.029741:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572921.029745:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488582272:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788572921.029904:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488582400:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788572921.029938:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572921.029959:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572921/real 1788572921] req@ffff94d408c61500 x1875450488582400/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572921.029978:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488582400/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572921.029989:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572921.029994:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572921.029998:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488582400:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788572921.030007:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488582528:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788572921.030031:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572921.030043:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572921/real 1788572921] req@ffff94d408c61c00 x1875450488582528/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572921.030056:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488582528/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572921.030072:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572921.030076:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572921.030095:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488582528:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788572921.030103:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488582656:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788572921.030127:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572921.030140:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572921/real 1788572921] req@ffff94d408c60e00 x1875450488582656/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572921.030153:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60e00 x1875450488582656/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572921.030161:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572921.030164:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572921.030168:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488582656:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572921.030175:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488582784:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788572921.030196:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572921.030241:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572921/real 1788572921] req@ffff94d408c63100 x1875450488582784/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572921.030255:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63100 x1875450488582784/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572921.030263:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572921.030266:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572921.030271:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488582784:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572921.541320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572921.989381:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572922.565311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572923.013355:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572923.589333:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572924.037396:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572924.613351:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572925.061420:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572925.637322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572926.085380:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572926.149316:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572926.149325:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572926.149328:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572926.149337:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4074 00000100:00080000:2.0:1788572926.149346:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572926.149352:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788572926.149384:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488582912/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572926.149407:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572926.149412:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572926.149413:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572926.149416:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4074 00000100:00080000:2.0:1788572926.149418:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572926.149420:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572926.149427:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572926.149435:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488583040/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572926.149442:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572926.149445:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572926.149446:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572926.149448:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4074 00000100:00080000:2.0:1788572926.149451:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572926.149453:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572926.149456:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572926.149464:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488583168/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572926.149471:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572926.149474:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572926.149475:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572926.149484:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4074 00000100:00080000:2.0:1788572926.149487:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572926.149490:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572926.149494:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572926.149504:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488582912:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788572926.149505:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63100 x1875450488583296/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572926.149533:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572926.149535:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572926.149535:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572926.149539:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4074 00000100:00080000:2.0:1788572926.149540:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572926.149543:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572926.149545:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572926.149554:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60e00 x1875450488583424/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788572926.149616:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572926.149665:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572926/real 1788572926] req@ffff94d408c63480 x1875450488582912/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572926.149691:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488582912/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572926.149705:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572926.149709:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572926.149715:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488582912:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788572926.149724:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488583040:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788572926.149749:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572926.149762:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572926/real 1788572926] req@ffff94d408c61880 x1875450488583040/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572926.149775:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488583040/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572926.149784:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572926.149787:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572926.149802:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488583040:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788572926.149814:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488583168:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788572926.149834:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572926.149846:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572926/real 1788572926] req@ffff94d408c61f80 x1875450488583168/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572926.149858:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488583168/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572926.149867:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572926.149871:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572926.149875:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488583168:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788572926.149909:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488583296:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788572926.149930:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572926.149941:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572926/real 1788572926] req@ffff94d408c63100 x1875450488583296/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572926.149954:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63100 x1875450488583296/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572926.149962:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572926.149966:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572926.149969:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488583296:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572926.149976:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488583424:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788572926.149993:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572926.150005:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572926/real 1788572926] req@ffff94d408c60e00 x1875450488583424/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572926.150017:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60e00 x1875450488583424/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572926.150025:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572926.150028:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572926.150032:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488583424:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572926.661350:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572927.109360:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572927.685291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572928.133443:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572928.709256:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572929.157335:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572929.733275:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572930.181486:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572930.757264:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572931.205337:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572931.269305:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572931.269313:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572931.269316:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572931.269324:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4079 00000100:00080000:3.0:1788572931.269334:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572931.269340:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788572931.269372:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488583552/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572931.269397:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572931.269400:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572931.269402:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572931.269404:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4079 00000100:00080000:3.0:1788572931.269406:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572931.269409:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572931.269416:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572931.269424:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488583680/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572931.269431:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572931.269434:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572931.269435:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572931.269438:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4079 00000100:00080000:3.0:1788572931.269440:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572931.269443:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572931.269446:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572931.269452:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488583808/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572931.269458:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572931.269460:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572931.269462:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788572931.269467:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488583552:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788572931.269467:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4079 00000100:00080000:3.0:1788572931.269470:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572931.269475:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572931.269480:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572931.269487:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488583936/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572931.269493:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572931.269496:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572931.269497:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572931.269500:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4079 00000100:00080000:3.0:1788572931.269503:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572931.269506:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572931.269511:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572931.269523:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488584064/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788572931.269546:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572931.269582:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572931/real 1788572931] req@ffff94d408c54380 x1875450488583552/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572931.269602:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488583552/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572931.269614:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572931.269618:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572931.269623:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488583552:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788572931.269631:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488583680:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572931.269656:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572931.269670:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572931/real 1788572931] req@ffff94d408c57100 x1875450488583680/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572931.269685:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488583680/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572931.269695:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572931.269699:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572931.269703:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488583680:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572931.269728:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488583808:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572931.269750:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572931.269763:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572931/real 1788572931] req@ffff94d408c56d80 x1875450488583808/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572931.269792:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488583808/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572931.269802:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572931.269806:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572931.269809:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488583808:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572931.269816:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488583936:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572931.269838:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572931.269851:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572931/real 1788572931] req@ffff94d408c57800 x1875450488583936/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572931.269865:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488583936/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572931.269874:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572931.269877:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572931.269881:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488583936:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572931.269888:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488584064:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572931.269908:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572931.269920:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572931/real 1788572931] req@ffff94d408c55c00 x1875450488584064/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572931.269933:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488584064/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572931.269948:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572931.269952:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572931.269955:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488584064:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572931.781305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572932.229349:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572932.805368:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572933.253390:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572933.829331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572934.277368:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572934.853318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572935.301464:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572935.877272:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572936.325360:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572936.389336:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572936.389344:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572936.389348:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572936.389356:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4084 00000100:00080000:1.0:1788572936.389365:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572936.389371:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788572936.389411:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488584192/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572936.389441:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572936.389445:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572936.389447:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572936.389451:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4084 00000100:00080000:1.0:1788572936.389455:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572936.389460:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572936.389471:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572936.389486:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488584320/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572936.389490:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488584192:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788572936.389506:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572936.389509:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572936.389510:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572936.389514:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4084 00000100:00080000:1.0:1788572936.389518:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572936.389521:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572936.389525:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572936.389536:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488584448/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572936.389542:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572936.389545:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572936.389546:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572936.389548:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4084 00000100:00080000:1.0:1788572936.389551:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572936.389553:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572936.389556:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788572936.389558:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572936.389566:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488584576/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572936.389573:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572936.389576:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572936.389577:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572936.389580:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4084 00000100:00080000:1.0:1788572936.389582:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572936.389585:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572936.389588:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572936.389594:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572936/real 1788572936] req@ffff94d4049bf800 x1875450488584192/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572936.389596:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdf80 x1875450488584704/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572936.389640:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488584192/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572936.389651:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572936.389654:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572936.389658:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488584192:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788572936.389678:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488584320:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572936.389694:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572936.389702:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572936/real 1788572936] req@ffff94d4049bed80 x1875450488584320/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572936.389710:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488584320/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572936.389729:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572936.389732:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572936.389734:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488584320:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572936.389738:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488584448:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572936.389755:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572936.389765:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572936/real 1788572936] req@ffff94d4049bc000 x1875450488584448/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572936.389779:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488584448/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572936.389788:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572936.389792:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572936.389795:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488584448:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572936.389802:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488584576:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572936.389823:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572936.389832:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572936/real 1788572936] req@ffff94d4049bfb80 x1875450488584576/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572936.389840:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488584576/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572936.389845:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572936.389848:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572936.389852:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488584576:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572936.389858:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488584704:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572936.389876:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572936.389887:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572936/real 1788572936] req@ffff94d4049bdf80 x1875450488584704/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572936.389901:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdf80 x1875450488584704/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572936.389909:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572936.389913:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572936.389916:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488584704:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572936.901364:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572937.349427:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572937.925315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572938.373400:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572938.949272:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572939.397452:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572939.973317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572940.421471:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572940.997313:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572941.445361:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572941.509275:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572941.509284:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572941.509288:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572941.509298:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4089 00000100:00080000:1.0:1788572941.509319:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572941.509326:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788572941.509363:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488584832/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572941.509384:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572941.509387:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572941.509388:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572941.509391:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4089 00000100:00080000:1.0:1788572941.509393:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572941.509401:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572941.509409:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572941.509418:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488584960/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572941.509426:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572941.509429:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572941.509432:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788572941.509433:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488584832:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788572941.509436:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4089 00000100:00080000:1.0:1788572941.509440:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572941.509444:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572941.509447:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572941.509455:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488585088/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572941.509462:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572941.509464:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572941.509466:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572941.509468:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4089 00000100:00080000:1.0:1788572941.509483:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572941.509486:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572941.509490:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572941.509503:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488585216/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788572941.509504:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788572941.509512:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572941.509515:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572941.509516:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572941.509519:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4089 00000100:00080000:1.0:1788572941.509522:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572941.509525:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572941.509528:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572941.509535:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488585344/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572941.509541:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572941/real 1788572941] req@ffff94d4049bea00 x1875450488584832/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572941.509558:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488584832/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572941.509566:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572941.509569:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572941.509572:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488584832:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788572941.509590:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488584960:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572941.509606:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572941.509614:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572941/real 1788572941] req@ffff94d4049bf480 x1875450488584960/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572941.509622:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488584960/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572941.509627:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572941.509629:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572941.509631:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488584960:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572941.509634:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488585088:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572941.509644:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572941.509649:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572941/real 1788572941] req@ffff94d4049bce00 x1875450488585088/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572941.509657:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488585088/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572941.509664:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572941.509668:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572941.509671:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488585088:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572941.509677:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488585216:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572941.509698:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572941.509710:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572941/real 1788572941] req@ffff94d4049bd180 x1875450488585216/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572941.509730:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488585216/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572941.509736:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572941.509738:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572941.509740:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488585216:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572941.509744:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488585344:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572941.509758:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572941.509765:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572941/real 1788572941] req@ffff94d4049bc380 x1875450488585344/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572941.509772:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc380 x1875450488585344/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572941.509777:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572941.509779:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572941.509781:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488585344:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572942.021295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572942.469377:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572943.045297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572943.493350:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572944.069297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572944.517380:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572945.093351:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572945.541486:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572946.117359:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572946.565361:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572946.629325:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572946.629337:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572946.629341:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572946.629351:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4094 00000100:00080000:3.0:1788572946.629362:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572946.629371:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788572946.629419:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488585472/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572946.629449:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572946.629453:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572946.629455:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572946.629459:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4094 00000100:00080000:3.0:1788572946.629463:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572946.629467:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572946.629477:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572946.629495:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488585472:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788572946.629495:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488585600/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572946.629513:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572946.629517:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572946.629519:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572946.629524:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4094 00000100:00080000:3.0:1788572946.629528:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572946.629532:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572946.629537:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572946.629553:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488585728/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572946.629570:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572946.629574:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572946.629577:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572946.629580:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4094 00000100:00080000:3.0:1788572946.629584:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572946.629588:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572946.629594:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788572946.629604:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572946.629610:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488585856/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572946.629622:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572946.629626:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572946.629628:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572946.629632:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4094 00000100:00080000:3.0:1788572946.629637:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572946.629642:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788572946.629648:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572946.629651:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572946/real 1788572946] req@ffff94d408c55880 x1875450488585472/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572946.629662:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488585984/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572946.629693:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488585472/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572946.629703:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572946.629707:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572946.629710:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488585472:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788572946.629730:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488585600:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572946.629749:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572946.629758:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572946/real 1788572946] req@ffff94d408c56a00 x1875450488585600/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572946.629766:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488585600/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572946.629772:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572946.629774:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572946.629776:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488585600:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572946.629780:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488585728:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572946.629790:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572946.629796:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572946/real 1788572946] req@ffff94d408c55180 x1875450488585728/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572946.629803:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488585728/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572946.629812:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572946.629815:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572946.629817:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488585728:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572946.629820:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488585856:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572946.629831:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572946.629836:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572946/real 1788572946] req@ffff94d408c54000 x1875450488585856/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572946.629843:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488585856/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572946.629848:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572946.629850:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572946.629852:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488585856:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572946.629855:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488585984:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572946.629864:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572946.629870:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572946/real 1788572946] req@ffff94d408c57480 x1875450488585984/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572946.629881:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488585984/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572946.629889:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572946.629892:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572946.629896:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488585984:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572947.141374:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572947.589517:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572948.165349:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572948.613409:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572949.189305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572949.637467:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572950.213276:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572950.661385:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572951.237338:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572951.685357:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572951.749334:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572951.749342:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572951.749346:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572951.749354:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4100 00000100:00080000:2.0:1788572951.749363:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572951.749369:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788572951.749413:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488586112/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572951.749441:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572951.749445:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572951.749448:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572951.749452:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4100 00000100:00080000:2.0:1788572951.749456:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572951.749461:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572951.749471:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572951.749486:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488586240/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572951.749489:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488586112:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788572951.749507:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572951.749511:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572951.749513:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572951.749519:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4100 00000100:00080000:2.0:1788572951.749523:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572951.749527:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572951.749532:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572951.749547:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488586368/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572951.749558:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572951.749562:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572951.749564:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572951.749568:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4100 00000100:00080000:2.0:1788572951.749572:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572951.749575:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572951.749580:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572951.749582:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488586240:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788572951.749592:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488586496/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572951.749607:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788572951.749610:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488586368:192.168.202.125@tcp:38:kworker.0 00000100:00080000:2.0:1788572951.749611:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572951.749614:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572951.749620:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4100 00000100:00080000:2.0:1788572951.749623:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572951.749626:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572951.749630:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572951.749641:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488586496:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788572951.749642:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488586624/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572951.749669:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488586624:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572952.261304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572952.709384:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572953.285335:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572953.733412:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572954.309398:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572954.757389:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788572954.821497:0:2392:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:1.0:1788572954.821544:0:2392:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788572954.821562:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788572954.821604:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788572954.821615:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788572954.821626:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788572954.821638:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:0.0:1788572954.821697:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572951/real 1788572954] req@ffff94d408c63800 x1875450488586624/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572954.821732:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488586624/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572954.821745:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572954.821751:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572954.821763:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488586624:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572954.821774:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572951/real 1788572954] req@ffff94d408c62a00 x1875450488586496/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572954.821788:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488586496/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572954.821802:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572954.821806:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572954.821810:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488586496:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572954.821819:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572951/real 1788572954] req@ffff94d408c61500 x1875450488586240/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572954.821835:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488586240/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572954.821845:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572954.821849:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572954.821853:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488586240:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572954.821862:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572951/real 1788572954] req@ffff94d408c63b80 x1875450488586368/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572954.821875:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488586368/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572954.821887:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572954.821891:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572954.821895:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488586368:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572954.821904:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572951/real 1788572954] req@ffff94d408c61c00 x1875450488586112/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572954.821917:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488586112/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572954.821926:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572954.821929:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572954.821933:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488586112:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788572955.333300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572955.781464:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572956.357345:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572956.805363:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572956.869428:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572956.869440:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572956.869445:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572956.869457:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4105 00000100:00080000:1.0:1788572956.869469:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572956.869478:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788572956.869526:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488586752/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572956.869559:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572956.869564:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572956.869566:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572956.869571:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4105 00000100:00080000:1.0:1788572956.869576:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572956.869581:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572956.869594:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572956.869610:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488586880/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572956.869628:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788572956.869629:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488586752:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788572956.869637:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572956.869642:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572956.869655:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4105 00000100:00080000:1.0:1788572956.869661:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572956.869667:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572956.869674:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572956.869693:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488587008/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572956.869707:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572956.869712:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572956.869714:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572956.869721:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4105 00000100:00080000:1.0:1788572956.869726:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572956.869732:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572956.869740:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788572956.869741:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572956.869769:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488587136/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572956.869789:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572956.869794:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572956.869796:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572956.869803:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4105 00000100:00100000:0.0:1788572956.869813:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572956/real 1788572956] req@ffff94d4049bc380 x1875450488586752/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572956.869814:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572956.869826:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572956.869837:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572956.869861:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488587264/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572956.869872:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc380 x1875450488586752/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572956.869936:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572956.869946:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572956.869955:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488586752:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788572956.869998:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488586880:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572956.870032:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572956.870054:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572956/real 1788572956] req@ffff94d4049bd180 x1875450488586880/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572956.870071:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488586880/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572956.870080:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572956.870085:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572956.870093:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488586880:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572956.870103:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488587008:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572956.870141:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572956.870153:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572956/real 1788572956] req@ffff94d4049bce00 x1875450488587008/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572956.870166:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488587008/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572956.870175:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572956.870179:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572956.870183:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488587008:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572956.870191:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488587136:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572956.870338:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572956.870361:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572956/real 1788572956] req@ffff94d4049bf480 x1875450488587136/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572956.870373:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488587136/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572956.870381:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572956.870385:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572956.870388:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488587136:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572956.870396:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488587264:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572956.870418:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572956.870430:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572956/real 1788572956] req@ffff94d4049bea00 x1875450488587264/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572956.870443:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488587264/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572956.870451:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572956.870455:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572956.870458:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488587264:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572957.381290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572957.829442:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572958.405345:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572958.853372:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572959.429366:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572959.877475:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572960.453920:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572960.901417:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572961.477353:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572961.925351:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572961.989298:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572961.989306:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572961.989309:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572961.989318:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4110 00000100:00080000:2.0:1788572961.989327:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572961.989333:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788572961.989365:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488587392/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572961.989388:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572961.989391:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572961.989393:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572961.989396:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4110 00000100:00080000:2.0:1788572961.989398:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572961.989401:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572961.989409:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572961.989417:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488587520/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572961.989425:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572961.989428:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572961.989429:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572961.989432:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4110 00000100:00080000:2.0:1788572961.989434:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572961.989436:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572961.989439:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572961.989446:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488587392:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788572961.989447:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488587648/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572961.989462:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572961.989464:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572961.989465:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572961.989468:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4110 00000100:00080000:2.0:1788572961.989469:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572961.989472:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572961.989475:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572961.989482:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488587776/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572961.989488:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572961.989491:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572961.989492:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572961.989495:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4110 00000100:00080000:2.0:1788572961.989498:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572961.989500:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572961.989503:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572961.989509:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488587904/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788572961.989523:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572961.989559:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572961/real 1788572961] req@ffff94d408c62680 x1875450488587392/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572961.989578:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488587392/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572961.989586:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572961.989589:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572961.989592:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488587392:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788572961.989597:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488587520:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572961.989621:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572961.989630:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572961/real 1788572961] req@ffff94d408c61180 x1875450488587520/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572961.989643:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488587520/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572961.989648:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572961.989651:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572961.989653:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488587520:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572961.989671:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488587648:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788572961.989682:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572961.989688:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572961/real 1788572961] req@ffff94d408c62d80 x1875450488587648/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572961.989696:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488587648/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572961.989701:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572961.989703:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572961.989705:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488587648:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788572961.989708:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488587776:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572961.989718:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572961.989724:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572961/real 1788572961] req@ffff94d408c60000 x1875450488587776/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572961.989731:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488587776/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572961.989740:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572961.989742:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572961.989744:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488587776:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572961.989747:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488587904:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788572961.989756:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572961.989761:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572961/real 1788572961] req@ffff94d408c60a80 x1875450488587904/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572961.989768:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488587904/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572961.989772:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572961.989774:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788572961.989776:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488587904:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788572962.501650:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572962.949341:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572963.525497:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572963.973353:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572964.549338:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572964.997385:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572965.574573:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572966.021373:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572966.597346:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572967.045388:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572967.109332:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572967.109343:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572967.109346:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572967.109356:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4115 00000100:00080000:0.0:1788572967.109365:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572967.109371:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788572967.109405:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488588032/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572967.109430:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572967.109433:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572967.109435:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572967.109437:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4115 00000100:00080000:0.0:1788572967.109440:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572967.109442:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572967.109450:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572967.109459:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488588160/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572967.109466:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572967.109469:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572967.109470:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572967.109472:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4115 00000100:00080000:0.0:1788572967.109474:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572967.109479:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788572967.109481:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488588032:192.168.202.125@tcp:250:kworker.0 00000008:00100000:0.0:1788572967.109483:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572967.109494:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488588288/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572967.109501:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572967.109503:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572967.109505:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572967.109508:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4115 00000100:00080000:0.0:1788572967.109510:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572967.109512:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572967.109515:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572967.109524:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488588416/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572967.109537:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572967.109539:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572967.109541:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572967.109543:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4115 00000100:00080000:0.0:1788572967.109545:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572967.109548:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572967.109555:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788572967.109555:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572967.109567:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488588544/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572967.109599:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572967/real 1788572967] req@ffff94d408d3ea00 x1875450488588032/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572967.109616:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488588032/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572967.109624:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572967.109626:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572967.109629:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488588032:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788572967.109635:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488588160:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788572967.109652:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572967.109661:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572967/real 1788572967] req@ffff94d408d3ce00 x1875450488588160/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572967.109669:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488588160/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572967.109675:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572967.109677:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572967.109679:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488588160:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788572967.109702:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488588288:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788572967.109722:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572967.109734:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572967/real 1788572967] req@ffff94d408d3df80 x1875450488588288/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572967.109748:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488588288/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572967.109754:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572967.109758:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572967.109761:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488588288:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788572967.109767:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488588416:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788572967.109789:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572967.109801:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572967/real 1788572967] req@ffff94d408d3d180 x1875450488588416/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572967.109814:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488588416/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572967.109823:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572967.109826:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572967.109830:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488588416:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572967.109836:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488588544:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788572967.109863:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572967.109871:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572967/real 1788572967] req@ffff94d408d3c000 x1875450488588544/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572967.109880:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488588544/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572967.109885:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572967.109887:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572967.109889:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488588544:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788572967.621310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572968.069347:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572968.645309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572969.093366:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572969.669285:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572970.117362:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572970.693314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572971.141373:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572971.717293:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572972.165377:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572972.229339:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572972.229351:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572972.229355:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572972.229367:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4120 00000100:00080000:1.0:1788572972.229378:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572972.229386:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788572972.229423:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdf80 x1875450488588672/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572972.229453:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572972.229457:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572972.229460:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572972.229464:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4120 00000100:00080000:1.0:1788572972.229468:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572972.229474:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572972.229485:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572972.229493:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488588672:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788572972.229501:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488588800/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572972.229522:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572972.229527:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572972.229529:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572972.229534:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4120 00000100:00080000:1.0:1788572972.229539:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572972.229543:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:3.0:1788572972.229548:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:1.0:1788572972.229549:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572972.229565:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488588928/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572972.229576:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572972.229580:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572972.229583:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572972.229589:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4120 00000100:00100000:3.0:1788572972.229589:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572972/real 1788572972] req@ffff94d4049bdf80 x1875450488588672/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572972.229596:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572972.229604:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572972.229612:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572972.229620:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdf80 x1875450488588672/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572972.229625:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488589056/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572972.229632:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572972.229641:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:1.0:1788572972.229644:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788572972.229648:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488588672:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788572972.229650:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572972.229654:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572972.229660:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4120 00000100:00080000:1.0:1788572972.229665:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572972.229669:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572972.229674:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572972.229679:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488588800:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572972.229686:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488589184/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788572972.229709:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572972.229722:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572972/real 1788572972] req@ffff94d4049bfb80 x1875450488588800/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572972.229731:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488588800/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572972.229746:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572972.229749:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572972.229753:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488588800:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788572972.229760:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488588928:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788572972.229790:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572972.229801:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572972/real 1788572972] req@ffff94d4049bc000 x1875450488588928/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572972.229810:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488588928/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572972.229816:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572972.229818:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572972.229820:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488588928:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788572972.229824:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488589056:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788572972.229839:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572972.229845:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572972/real 1788572972] req@ffff94d4049bed80 x1875450488589056/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572972.229853:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488589056/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572972.229857:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572972.229859:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572972.229862:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488589056:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572972.229873:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488589184:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788572972.229883:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572972.229889:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572972/real 1788572972] req@ffff94d4049bf800 x1875450488589184/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572972.229897:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488589184/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572972.229902:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572972.229903:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572972.229905:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488589184:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788572972.741273:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572973.189366:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572973.765305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572974.213373:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572974.789283:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572975.237390:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572975.813277:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572976.261404:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572976.837310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572977.285441:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572977.349376:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572977.349387:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572977.349392:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572977.349403:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4125 00000100:00080000:2.0:1788572977.349417:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572977.349424:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788572977.349470:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62300 x1875450488589312/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572977.349497:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572977.349501:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572977.349503:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572977.349507:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4125 00000100:00080000:2.0:1788572977.349510:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572977.349515:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572977.349527:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572977.349544:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60380 x1875450488589440/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572977.349555:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572977.349558:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572977.349560:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572977.349563:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4125 00000100:00080000:2.0:1788572977.349566:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572977.349570:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572977.349574:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572977.349585:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488589312:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788572977.349586:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60700 x1875450488589568/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572977.349607:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572977.349611:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572977.349612:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572977.349619:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4125 00000100:00080000:2.0:1788572977.349623:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572977.349626:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572977.349630:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572977.349643:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488589696/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572977.349656:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572977.349660:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572977.349665:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788572977.349666:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788572977.349677:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4125 00000100:00080000:2.0:1788572977.349681:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572977.349684:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572977.349688:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572977.349700:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488589824/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572977.349728:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572977/real 1788572977] req@ffff94d408c62300 x1875450488589312/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572977.349749:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62300 x1875450488589312/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572977.349758:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572977.349761:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572977.349765:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488589312:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788572977.349774:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488589440:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788572977.349810:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572977.349826:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572977/real 1788572977] req@ffff94d408c60380 x1875450488589440/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572977.349835:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60380 x1875450488589440/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572977.349841:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572977.349843:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572977.349846:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488589440:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788572977.349874:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488589568:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788572977.349897:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572977.349909:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572977/real 1788572977] req@ffff94d408c60700 x1875450488589568/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572977.349918:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60700 x1875450488589568/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572977.349924:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572977.349926:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572977.349928:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488589568:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788572977.349932:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488589696:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788572977.349946:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572977.349953:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572977/real 1788572977] req@ffff94d408c60a80 x1875450488589696/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572977.349960:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488589696/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572977.349966:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572977.349968:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572977.349969:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488589696:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572977.349973:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488589824:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788572977.349983:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572977.349988:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572977/real 1788572977] req@ffff94d408c60000 x1875450488589824/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572977.349995:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488589824/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572977.350000:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572977.350002:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572977.350004:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488589824:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788572977.861318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572978.309440:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572978.885324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572979.333372:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572979.909297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572980.357396:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572980.933328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572981.381423:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572981.957301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572982.405436:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572982.469346:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572982.469357:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572982.469362:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572982.469373:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4130 00000100:00080000:2.0:1788572982.469387:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572982.469396:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788572982.469447:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488589952/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572982.469483:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572982.469486:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572982.469487:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572982.469497:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4130 00000100:00080000:2.0:1788572982.469500:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572982.469506:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572982.469517:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572982.469528:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488590080/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572982.469537:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572982.469542:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788572982.469542:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488589952:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788572982.469545:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572982.469553:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4130 00000100:00080000:2.0:1788572982.469556:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572982.469558:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572982.469561:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572982.469571:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488590208/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572982.469579:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572982.469581:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572982.469583:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572982.469586:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4130 00000100:00080000:2.0:1788572982.469588:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572982.469591:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572982.469594:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572982.469601:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488590336/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572982.469607:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572982.469609:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572982.469610:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572982.469612:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4130 00000100:00080000:2.0:1788572982.469615:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572982.469617:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788572982.469620:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788572982.469626:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788572982.469629:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488590464/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572982.469670:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572982/real 1788572982] req@ffff94d408c62d80 x1875450488589952/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572982.469687:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488589952/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572982.469696:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572982.469700:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572982.469703:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488589952:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788572982.469726:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488590080:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788572982.469742:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572982.469751:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572982/real 1788572982] req@ffff94d408c61180 x1875450488590080/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572982.469759:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488590080/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572982.469765:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572982.469767:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572982.469770:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488590080:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788572982.469773:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488590208:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788572982.469783:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572982.469788:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572982/real 1788572982] req@ffff94d408c62680 x1875450488590208/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572982.469796:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488590208/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572982.469802:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572982.469803:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572982.469805:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488590208:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788572982.469809:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488590336:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788572982.469820:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572982.469826:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572982/real 1788572982] req@ffff94d408c61c00 x1875450488590336/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572982.469834:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488590336/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572982.469839:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572982.469841:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572982.469842:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488590336:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572982.469846:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488590464:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788572982.469855:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572982.469860:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572982/real 1788572982] req@ffff94d408c63b80 x1875450488590464/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572982.469868:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488590464/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572982.469873:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572982.469875:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572982.469876:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488590464:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788572982.981365:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572983.429450:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572984.005358:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572984.453461:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572985.029372:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572985.477418:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572986.053359:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572986.501376:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572987.077303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572987.525343:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572987.589320:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572987.589332:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572987.589337:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572987.589345:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4135 00000100:00080000:0.0:1788572987.589354:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572987.589360:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788572987.589392:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488590592/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572987.589415:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572987.589418:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572987.589420:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572987.589423:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4135 00000100:00080000:0.0:1788572987.589425:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572987.589427:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572987.589434:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572987.589442:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488590720/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572987.589453:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572987.589457:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572987.589459:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788572987.589464:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488590592:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788572987.589466:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4135 00000100:00080000:0.0:1788572987.589474:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572987.589480:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572987.589486:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572987.589507:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488590848/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572987.589516:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572987.589519:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572987.589521:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572987.589524:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4135 00000100:00080000:0.0:1788572987.589527:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572987.589529:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572987.589534:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788572987.589535:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788572987.589546:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488590976/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572987.589554:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572987.589556:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572987.589558:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572987.589560:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4135 00000100:00080000:0.0:1788572987.589563:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572987.589565:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572987.589568:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572987.589575:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488591104/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572987.589583:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572987/real 1788572987] req@ffff94d408d3dc00 x1875450488590592/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572987.589604:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488590592/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572987.589617:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572987.589620:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572987.589623:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488590592:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788572987.589628:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488590720:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788572987.589657:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572987.589667:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572987/real 1788572987] req@ffff94d408d3c380 x1875450488590720/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572987.589676:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488590720/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572987.589681:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572987.589683:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572987.589685:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488590720:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788572987.589703:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488590848:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788572987.589722:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572987.589733:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572987/real 1788572987] req@ffff94d408d3c700 x1875450488590848/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572987.589746:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488590848/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572987.589764:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572987.589767:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572987.589770:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488590848:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788572987.589775:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488590976:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788572987.589792:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572987.589800:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572987/real 1788572987] req@ffff94d408d3e300 x1875450488590976/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572987.589810:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488590976/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572987.589818:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572987.589828:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572987.589832:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488590976:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572987.589838:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488591104:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788572987.589859:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572987.589872:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572987/real 1788572987] req@ffff94d408d3d880 x1875450488591104/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572987.589884:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488591104/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572987.589891:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572987.589894:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572987.589905:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488591104:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788572988.101320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572988.549368:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572989.125282:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572989.573347:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572990.149295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572990.597357:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572991.173327:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572991.621412:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572992.197362:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572992.645397:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572992.709313:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572992.709322:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572992.709325:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572992.709335:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4141 00000100:00080000:0.0:1788572992.709345:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572992.709351:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788572992.709387:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ed80 x1875450488591232/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572992.709421:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572992.709425:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572992.709427:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572992.709430:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4141 00000100:00080000:0.0:1788572992.709433:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572992.709436:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572992.709444:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572992.709455:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488591360/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572992.709464:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572992.709468:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788572992.709468:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488591232:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788572992.709471:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572992.709479:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4141 00000100:00080000:0.0:1788572992.709485:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572992.709490:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572992.709496:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572992.709513:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488591488/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572992.709532:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788572992.709536:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788572992.709539:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572992.709542:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572992.709548:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4141 00000100:00080000:0.0:1788572992.709553:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572992.709560:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788572992.709566:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572992.709583:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488591616/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572992.709588:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572992/real 1788572992] req@ffff94d408d3ed80 x1875450488591232/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572992.709597:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572992.709603:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572992.709605:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572992.709611:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4141 00000100:00100000:1.0:1788572992.709613:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ed80 x1875450488591232/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572992.709615:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572992.709625:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572992.709632:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00080000:1.0:1788572992.709633:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572992.709641:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488591232:192.168.202.125@tcp:250:kworker.0 00000008:00100000:0.0:1788572992.709642:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572992.709655:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488591744/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572992.709673:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488591360:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572992.709704:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572992.709720:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572992/real 1788572992] req@ffff94d408d3f480 x1875450488591360/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572992.709747:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488591360/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572992.709756:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572992.709760:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572992.709763:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488591360:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572992.709769:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488591488:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788572992.709791:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572992.709804:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572992/real 1788572992] req@ffff94d408d3d880 x1875450488591488/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572992.709816:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488591488/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572992.709832:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572992.709837:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572992.709841:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488591488:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788572992.709849:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488591616:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572992.709873:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572992.709882:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572992/real 1788572992] req@ffff94d408d3e300 x1875450488591616/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572992.709896:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488591616/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572992.709902:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572992.709904:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572992.709906:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488591616:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788572992.709910:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488591744:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788572992.709924:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788572992.709931:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572992/real 1788572992] req@ffff94d408d3c700 x1875450488591744/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572992.709939:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488591744/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572992.709944:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572992.709946:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788572992.709949:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488591744:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788572993.221331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572993.669347:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572994.245270:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572994.693329:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572995.269316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572995.717381:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572996.293297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572996.741413:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572997.317322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572997.765359:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572997.829320:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572997.829329:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572997.829332:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572997.829341:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4146 00000100:00080000:1.0:1788572997.829351:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572997.829357:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788572997.829389:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488591872/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572997.829423:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572997.829427:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572997.829428:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572997.829431:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4146 00000100:00080000:1.0:1788572997.829433:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572997.829436:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572997.829444:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572997.829453:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488592000/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572997.829465:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788572997.829465:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488591872:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788572997.829473:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572997.829476:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572997.829479:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4146 00000100:00080000:1.0:1788572997.829481:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572997.829483:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572997.829487:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572997.829496:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488592128/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572997.829503:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572997.829505:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572997.829506:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572997.829509:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4146 00000100:00080000:1.0:1788572997.829512:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572997.829515:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788572997.829518:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572997.829524:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488592256/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572997.829531:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572997.829533:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572997.829534:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572997.829536:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4146 00000100:00080000:1.0:1788572997.829539:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572997.829542:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:3.0:1788572997.829547:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:1.0:1788572997.829548:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572997.829558:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488592384/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572997.829588:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572997/real 1788572997] req@ffff94d4049bd500 x1875450488591872/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572997.829605:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488591872/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572997.829613:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572997.829616:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572997.829618:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488591872:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788572997.829637:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488592000:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788572997.829654:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572997.829662:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572997/real 1788572997] req@ffff94d4049bf100 x1875450488592000/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572997.829671:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488592000/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572997.829676:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572997.829678:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572997.829680:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488592000:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788572997.829684:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488592128:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788572997.829694:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572997.829699:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572997/real 1788572997] req@ffff94d4049bc700 x1875450488592128/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572997.829707:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488592128/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572997.829717:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572997.829719:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572997.829720:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488592128:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788572997.829724:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488592256:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788572997.829734:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572997.829739:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572997/real 1788572997] req@ffff94d4049bca80 x1875450488592256/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572997.829746:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488592256/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572997.829751:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572997.829753:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572997.829755:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488592256:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788572997.829758:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488592384:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788572997.829767:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788572997.829772:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572997/real 1788572997] req@ffff94d4049be680 x1875450488592384/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572997.829779:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488592384/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572997.829784:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572997.829786:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788572997.829789:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488592384:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788572998.341310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788572998.789382:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572999.365321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572999.813384:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573000.389313:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573000.837392:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573001.413324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573001.861456:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573002.437367:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573002.885402:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573002.949332:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573002.949343:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573002.949348:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573002.949359:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4151 00000100:00080000:2.0:1788573002.949372:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573002.949380:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573002.949433:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488592512/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573002.949471:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573002.949476:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573002.949478:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573002.949483:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4151 00000100:00080000:2.0:1788573002.949487:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573002.949492:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573002.949503:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573002.949518:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488592640/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573002.949528:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488592512:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788573002.949546:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573002.949549:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573002.949551:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573002.949555:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4151 00000100:00080000:2.0:1788573002.949558:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573002.949561:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573002.949566:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573002.949577:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488592768/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573002.949584:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573002.949587:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573002.949588:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573002.949590:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4151 00000100:00080000:2.0:1788573002.949600:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573002.949603:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573002.949607:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788573002.949611:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573002.949619:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60e00 x1875450488592896/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573002.949627:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573002.949629:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573002.949631:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573002.949633:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4151 00000100:00080000:2.0:1788573002.949638:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573002.949640:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573002.949643:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573002.949650:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63100 x1875450488593024/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573002.949658:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573002/real 1788573002] req@ffff94d408c61500 x1875450488592512/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573002.949678:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488592512/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573002.949687:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573002.949691:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573002.949695:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488592512:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573002.949718:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488592640:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573002.949734:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573002.949743:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573002/real 1788573002] req@ffff94d408c62a00 x1875450488592640/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573002.949751:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488592640/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573002.949757:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573002.949760:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573002.949762:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488592640:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573002.949766:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488592768:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573002.949776:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573002.949783:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573002/real 1788573002] req@ffff94d408c63800 x1875450488592768/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573002.949791:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488592768/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573002.949796:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573002.949798:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573002.949800:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488592768:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573002.949804:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488592896:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573002.949815:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573002.949821:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573002/real 1788573002] req@ffff94d408c60e00 x1875450488592896/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573002.949829:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60e00 x1875450488592896/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573002.949834:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573002.949836:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573002.949838:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488592896:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573002.949842:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488593024:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573002.949851:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573002.949857:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573002/real 1788573002] req@ffff94d408c63100 x1875450488593024/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573002.949864:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63100 x1875450488593024/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573002.949870:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573002.949872:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573002.949874:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488593024:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573003.461349:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573003.909432:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573004.485337:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573004.933336:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573005.509297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573005.957451:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573006.533332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573006.981418:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573007.557314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573008.005431:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573008.069367:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573008.069381:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573008.069387:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573008.069401:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4156 00000100:00080000:3.0:1788573008.069415:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573008.069425:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573008.069472:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488593152/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573008.069511:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573008.069517:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573008.069519:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573008.069523:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4156 00000100:00080000:3.0:1788573008.069527:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573008.069532:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573008.069544:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573008.069557:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488593280/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573008.069571:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573008.069575:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788573008.069581:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488593152:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573008.069581:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573008.069589:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4156 00000100:00080000:3.0:1788573008.069594:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573008.069598:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573008.069603:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573008.069615:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488593408/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573008.069626:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573008.069630:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573008.069633:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573008.069637:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4156 00000100:00080000:3.0:1788573008.069641:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573008.069646:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573008.069652:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573008.069666:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488593536/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573008.069678:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573008.069678:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573008.069685:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573008.069687:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573008.069693:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4156 00000100:00080000:3.0:1788573008.069697:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573008.069702:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573008.069707:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573008.069719:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488593664/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573008.069743:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573008/real 1788573008] req@ffff94d408c54700 x1875450488593152/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573008.069777:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488593152/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573008.069790:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573008.069795:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573008.069800:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488593152:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573008.069838:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488593280:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573008.069864:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573008.069878:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573008/real 1788573008] req@ffff94d408c55f80 x1875450488593280/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573008.069892:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488593280/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573008.069901:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573008.069905:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573008.069908:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488593280:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573008.069915:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488593408:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573008.069937:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573008.069950:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573008/real 1788573008] req@ffff94d408c56300 x1875450488593408/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573008.069963:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488593408/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573008.069979:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573008.069981:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573008.069984:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488593408:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573008.069988:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488593536:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573008.070005:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573008.070013:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573008/real 1788573008] req@ffff94d408c57480 x1875450488593536/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573008.070021:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488593536/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573008.070026:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573008.070028:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573008.070030:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488593536:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573008.070034:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488593664:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573008.070071:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573008.070080:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573008/real 1788573008] req@ffff94d408c54000 x1875450488593664/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573008.070088:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488593664/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573008.070093:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573008.070095:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573008.070098:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488593664:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573008.581272:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573009.029365:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573009.605300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573010.053395:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573010.629289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573011.077384:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573011.653294:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573012.101394:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573012.677374:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573013.125436:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573013.189281:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573013.189289:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573013.189292:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573013.189300:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4161 00000100:00080000:3.0:1788573013.189309:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573013.189315:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573013.189347:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488593792/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573013.189373:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573013.189377:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573013.189378:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573013.189382:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4161 00000100:00080000:3.0:1788573013.189386:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573013.189390:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573013.189400:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573013.189412:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488593920/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573013.189423:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488593792:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573013.189423:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573013.189426:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573013.189428:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573013.189431:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4161 00000100:00080000:3.0:1788573013.189434:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573013.189438:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573013.189442:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573013.189453:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488594048/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573013.189461:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573013.189464:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573013.189465:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573013.189468:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4161 00000100:00080000:3.0:1788573013.189472:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573013.189475:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573013.189480:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573013.189483:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573013.189488:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488594176/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573013.189496:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573013.189498:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573013.189500:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573013.189503:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4161 00000100:00080000:3.0:1788573013.189506:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573013.189509:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573013.189512:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573013.189516:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573013/real 1788573013] req@ffff94d408c55180 x1875450488593792/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573013.189521:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488594304/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573013.189534:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488593792/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573013.189545:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573013.189548:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573013.189553:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488593792:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573013.189576:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488593920:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573013.189595:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573013.189604:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573013/real 1788573013] req@ffff94d408c56a00 x1875450488593920/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573013.189612:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488593920/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573013.189618:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573013.189620:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573013.189622:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488593920:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573013.189628:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488594048:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573013.189638:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573013.189643:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573013/real 1788573013] req@ffff94d408c55880 x1875450488594048/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573013.189651:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488594048/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573013.189656:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573013.189657:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573013.189659:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488594048:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573013.189663:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488594176:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573013.189676:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573013.189685:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573013/real 1788573013] req@ffff94d408c55c00 x1875450488594176/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573013.189696:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488594176/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573013.189705:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573013.189708:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573013.189712:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488594176:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573013.189719:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488594304:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573013.189736:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573013.189744:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573013/real 1788573013] req@ffff94d408c57800 x1875450488594304/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573013.189753:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488594304/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573013.189758:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573013.189760:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573013.189762:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488594304:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573013.701292:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573014.149340:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573014.725316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573015.173368:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573015.749269:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573016.197337:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573016.773306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573017.221357:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573017.797333:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573018.245366:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573018.309347:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573018.309362:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573018.309368:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573018.309382:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4166 00000100:00080000:3.0:1788573018.309396:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573018.309405:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573018.309458:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488594432/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573018.309495:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573018.309500:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573018.309502:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573018.309507:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4166 00000100:00080000:3.0:1788573018.309511:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573018.309516:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573018.309527:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573018.309543:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488594560/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573018.309553:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573018.309557:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488594432:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573018.309557:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573018.309559:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573018.309563:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4166 00000100:00080000:3.0:1788573018.309567:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573018.309571:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573018.309575:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573018.309587:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488594688/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573018.309597:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573018.309600:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573018.309602:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573018.309605:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4166 00000100:00080000:3.0:1788573018.309608:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573018.309612:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573018.309616:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573018.309626:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488594816/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573018.309635:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573018.309638:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573018.309640:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573018.309644:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4166 00000100:00080000:3.0:1788573018.309647:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573018.309651:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573018.309656:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573018.309666:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488594944/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573018.309683:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488594560:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573018.309728:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488594688:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573018.309752:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488594816:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573018.309790:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488594944:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573018.821383:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573019.269509:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573019.845307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573020.293394:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573020.869321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573021.317399:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788573021.381495:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:2.0:1788573021.381543:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788573021.381562:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788573021.381611:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788573021.381623:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788573021.381635:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788573021.381645:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:1.0:1788573021.381765:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573018/real 1788573021] req@ffff94d408c57100 x1875450488594560/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573021.381794:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488594560/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573021.381836:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573021.381841:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573021.381847:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488594560:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573021.381857:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573018/real 1788573021] req@ffff94d408c54380 x1875450488594688/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573021.381875:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488594688/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573021.381883:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573021.381886:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573021.381890:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488594688:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573021.381896:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573018/real 1788573021] req@ffff94d408c54e00 x1875450488594816/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573021.381907:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488594816/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573021.381914:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573021.381918:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573021.381922:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488594816:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573021.381928:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573018/real 1788573021] req@ffff94d408c57b80 x1875450488594944/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573021.381952:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488594944/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573021.381959:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573021.381963:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573021.381966:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488594944:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573021.381973:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573018/real 1788573021] req@ffff94d408c56d80 x1875450488594432/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573021.381984:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488594432/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573021.381991:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573021.381994:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573021.381997:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488594432:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788573021.893327:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573022.341371:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573022.917280:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573023.365362:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573023.429372:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573023.429383:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573023.429387:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573023.429398:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4171 00000100:00080000:3.0:1788573023.429409:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573023.429416:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573023.429448:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488595072/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573023.429480:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573023.429484:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573023.429485:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573023.429489:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4171 00000100:00080000:3.0:1788573023.429493:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573023.429497:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573023.429506:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573023.429518:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55500 x1875450488595200/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573023.429530:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573023.429534:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573023.429536:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573023.429542:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4171 00000100:00100000:1.0:1788573023.429544:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488595072:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573023.429546:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573023.429553:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573023.429559:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573023.429574:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488595328/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573023.429585:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573023.429589:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573023.429591:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573023.429595:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4171 00000100:00080000:3.0:1788573023.429605:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573023.429611:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573023.429616:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573023.429631:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488595456/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573023.429636:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573023.429647:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573023.429651:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573023.429654:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573023.429659:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4171 00000100:00080000:3.0:1788573023.429664:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573023.429668:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573023.429674:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573023.429680:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573023/real 1788573023] req@ffff94d408c56680 x1875450488595072/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573023.429689:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488595584/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573023.429720:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488595072/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573023.429733:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573023.429738:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573023.429743:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488595072:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573023.429759:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488595200:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573023.429817:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573023.429841:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573023/real 1788573023] req@ffff94d408c55500 x1875450488595200/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573023.429863:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55500 x1875450488595200/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573023.429872:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573023.429876:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573023.429880:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488595200:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573023.429907:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488595328:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573023.429932:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573023.429944:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573023/real 1788573023] req@ffff94d408c54a80 x1875450488595328/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573023.429957:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488595328/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573023.429966:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573023.429970:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573023.429974:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488595328:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573023.429981:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488595456:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573023.430002:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573023.430014:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573023/real 1788573023] req@ffff94d408c56d80 x1875450488595456/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573023.430025:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488595456/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573023.430038:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573023.430041:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573023.430044:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488595456:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573023.430050:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488595584:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573023.430068:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573023.430079:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573023/real 1788573023] req@ffff94d408c57b80 x1875450488595584/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573023.430090:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488595584/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573023.430097:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573023.430100:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573023.430103:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488595584:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573023.941283:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573024.389340:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573024.965333:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573025.413379:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573025.989327:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573026.437395:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573027.013870:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573027.461412:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573028.037816:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573028.485376:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573028.549363:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573028.549372:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573028.549379:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573028.549390:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4176 00000100:00080000:3.0:1788573028.549400:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573028.549409:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573028.549446:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488595712/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573028.549478:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573028.549482:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573028.549484:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573028.549487:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4176 00000100:00080000:3.0:1788573028.549491:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573028.549494:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573028.549502:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573028.549520:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488595840/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573028.549530:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488595712:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573028.549538:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573028.549543:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573028.549546:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573028.549552:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4176 00000100:00080000:3.0:1788573028.549557:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573028.549560:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573028.549565:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573028.549576:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488595968/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573028.549584:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788573028.549588:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573028.549589:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573028.549593:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573028.549597:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4176 00000100:00080000:3.0:1788573028.549660:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573028.549678:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573028.549684:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573028.549687:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573028/real 1788573028] req@ffff94d408c54e00 x1875450488595712/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573028.549701:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488596096/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573028.549705:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488595712/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573028.549713:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573028.549721:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573028.549725:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573028.549732:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4176 00000100:00080000:2.0:1788573028.549733:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573028.549736:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573028.549739:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:3.0:1788573028.549743:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573028.549747:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488595712:192.168.202.125@tcp:250:kworker.0 00000008:00100000:3.0:1788573028.549750:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573028.549769:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488595840:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788573028.549788:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573028.549798:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573028/real 1788573028] req@ffff94d408c54380 x1875450488595840/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573028.549807:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488595840/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573028.549812:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573028.549815:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573028.549817:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488595840:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788573028.549821:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488595968:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788573028.549833:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573028.549840:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573028/real 1788573028] req@ffff94d408c57100 x1875450488595968/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573028.549849:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488595968/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573028.549855:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573028.549857:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573028.549860:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488595968:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788573028.549863:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488596096:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573028.549867:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488596224/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573028.549877:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573028.549890:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573028/real 1788573028] req@ffff94d408c57800 x1875450488596096/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573028.549904:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488596096/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573028.549910:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573028.549913:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573028.549917:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488596096:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573028.549929:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488596224:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788573028.549940:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573028.549947:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573028/real 1788573028] req@ffff94d408c55c00 x1875450488596224/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573028.549959:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488596224/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573028.549965:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573028.549969:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573028.549972:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488596224:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573029.061974:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573029.509343:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573030.085655:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573030.533372:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573031.109520:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573031.557358:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573032.133413:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573032.581300:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573033.157320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573033.605312:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573033.669311:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573033.669321:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573033.669326:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573033.669336:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4181 00000100:00080000:3.0:1788573033.669347:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573033.669365:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573033.669403:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488596352/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573033.669432:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573033.669439:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573033.669442:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573033.669447:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4181 00000100:00080000:3.0:1788573033.669451:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573033.669456:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573033.669465:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488596352:192.168.202.125@tcp:250:kworker.0 00000008:00100000:3.0:1788573033.669466:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573033.669487:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488596480/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573033.669500:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573033.669504:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573033.669506:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573033.669510:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4181 00000100:00080000:3.0:1788573033.669514:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573033.669518:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:2.0:1788573033.669523:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:3.0:1788573033.669525:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573033.669542:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488596608/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573033.669553:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573033.669556:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1788573033.669562:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573033/real 1788573033] req@ffff94d408c55880 x1875450488596352/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573033.669562:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573033.669584:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4181 00000100:00080000:3.0:1788573033.669593:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573033.669599:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573033.669606:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573033.669610:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488596352/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573033.669621:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788573033.669623:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488596736/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573033.669628:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573033.669637:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488596352:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573033.669644:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573033.669656:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573033.669659:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573033.669664:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4181 00000100:00100000:2.0:1788573033.669666:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488596480:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788573033.669670:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573033.669679:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573033.669685:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573033.669701:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488596864/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573033.669709:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573033.669728:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573033/real 1788573033] req@ffff94d408c56a00 x1875450488596480/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573033.669748:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488596480/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573033.669755:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573033.669758:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573033.669760:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488596480:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788573033.669765:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488596608:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788573033.669780:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573033.669787:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573033/real 1788573033] req@ffff94d408c55180 x1875450488596608/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573033.669795:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488596608/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573033.669800:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573033.669802:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573033.669804:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488596608:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788573033.669808:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488596736:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788573033.669818:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573033.669824:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573033/real 1788573033] req@ffff94d408c54000 x1875450488596736/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573033.669836:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488596736/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573033.669841:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573033.669842:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573033.669851:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488596736:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573033.669863:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488596864:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788573033.669874:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573033.669881:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573033/real 1788573033] req@ffff94d408c57480 x1875450488596864/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573033.669889:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488596864/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573033.669894:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573033.669897:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573033.669899:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488596864:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573034.181303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573034.629366:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573035.205273:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573035.653378:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573036.229304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573036.677380:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573037.253261:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573037.701407:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573038.277326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573038.725389:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573038.789261:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573038.789269:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573038.789273:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573038.789281:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4187 00000100:00080000:3.0:1788573038.789290:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573038.789297:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573038.789330:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488596992/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573038.789361:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573038.789365:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573038.789367:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573038.789372:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4187 00000100:00080000:3.0:1788573038.789376:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573038.789381:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573038.789390:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573038.789402:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488597120/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573038.789411:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788573038.789412:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488596992:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573038.789416:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573038.789419:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573038.789425:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4187 00000100:00080000:3.0:1788573038.789427:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573038.789437:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573038.789446:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573038.789462:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488597248/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573038.789472:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573038.789475:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573038.789487:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573038.789494:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4187 00000800:00000100:2.0:1788573038.789496:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573038.789500:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573038.789505:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573038.789511:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573038.789521:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488597376/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573038.789533:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573038.789536:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1788573038.789538:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573038/real 1788573038] req@ffff94d408c56300 x1875450488596992/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573038.789538:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573038.789545:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4187 00000100:00080000:3.0:1788573038.789549:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573038.789554:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573038.789559:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573038.789568:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488597504/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573038.789578:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488596992/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573038.789591:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573038.789605:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573038.789610:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488596992:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788573038.789636:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488597120:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788573038.789663:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573038.789677:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573038/real 1788573038] req@ffff94d408c55f80 x1875450488597120/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573038.789692:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488597120/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573038.789701:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573038.789705:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573038.789709:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488597120:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788573038.789715:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488597248:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788573038.789734:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573038.789745:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573038/real 1788573038] req@ffff94d408c54700 x1875450488597248/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573038.789758:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488597248/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573038.789773:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573038.789776:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573038.789780:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488597248:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788573038.789786:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488597376:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788573038.789816:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573038.789828:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573038/real 1788573038] req@ffff94d408c57480 x1875450488597376/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573038.789840:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488597376/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573038.789848:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573038.789851:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573038.789855:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488597376:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573038.789861:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488597504:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788573038.789879:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573038.789890:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573038/real 1788573038] req@ffff94d408c54000 x1875450488597504/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573038.789902:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488597504/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573038.789910:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573038.789913:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573038.789917:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488597504:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573039.301305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573039.749371:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573040.325300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573040.773384:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573041.349284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573041.797378:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573042.373321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573042.821396:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573043.397303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573043.845354:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573043.909304:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573043.909312:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573043.909316:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573043.909325:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4192 00000100:00080000:0.0:1788573043.909334:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573043.909341:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573043.909374:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488597632/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573043.909414:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573043.909419:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573043.909420:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573043.909423:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4192 00000100:00080000:0.0:1788573043.909426:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573043.909429:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573043.909437:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573043.909447:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488597760/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573043.909456:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573043.909459:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573043.909461:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788573043.909464:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488597632:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788573043.909465:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4192 00000100:00080000:0.0:1788573043.909469:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573043.909474:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573043.909477:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573043.909485:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488597888/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573043.909491:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573043.909493:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573043.909495:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573043.909498:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4192 00000100:00080000:0.0:1788573043.909500:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573043.909502:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573043.909507:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573043.909513:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488598016/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573043.909519:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573043.909522:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573043.909523:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573043.909527:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4192 00000800:00000100:2.0:1788573043.909528:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788573043.909531:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573043.909534:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573043.909538:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573043.909546:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488598144/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573043.909565:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573043/real 1788573043] req@ffff94d408d3c380 x1875450488597632/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573043.909584:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488597632/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573043.909597:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573043.909601:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573043.909605:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488597632:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788573043.909627:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488597760:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788573043.909646:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573043.909655:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573043/real 1788573043] req@ffff94d408d3dc00 x1875450488597760/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573043.909663:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488597760/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573043.909669:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573043.909671:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573043.909674:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488597760:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788573043.909678:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488597888:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788573043.909688:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573043.909694:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573043/real 1788573043] req@ffff94d408d3c000 x1875450488597888/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573043.909701:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488597888/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573043.909706:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573043.909708:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573043.909709:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488597888:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788573043.909713:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488598016:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788573043.909723:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573043.909728:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573043/real 1788573043] req@ffff94d408d3d180 x1875450488598016/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573043.909735:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488598016/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573043.909740:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573043.909741:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573043.909743:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488598016:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573043.909746:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488598144:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788573043.909755:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573043.909761:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573043/real 1788573043] req@ffff94d408d3df80 x1875450488598144/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573043.909768:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488598144/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573043.909773:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573043.909775:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573043.909776:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488598144:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573044.421304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573044.869348:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573045.445286:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573045.893338:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573046.469321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573046.917390:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573047.493286:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573047.941346:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573048.517289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573048.965366:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573049.029351:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573049.029359:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573049.029363:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573049.029372:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4197 00000100:00080000:1.0:1788573049.029391:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573049.029404:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573049.029441:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd880 x1875450488598272/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573049.029465:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573049.029468:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573049.029473:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573049.029475:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4197 00000100:00080000:1.0:1788573049.029478:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573049.029482:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573049.029490:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573049.029499:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488598400/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573049.029507:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573049.029511:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788573049.029511:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488598272:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788573049.029515:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573049.029522:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4197 00000100:00080000:1.0:1788573049.029524:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573049.029526:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573049.029529:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573049.029540:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488598528/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573049.029547:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573049.029550:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573049.029551:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573049.029554:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4197 00000100:00080000:1.0:1788573049.029556:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573049.029559:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573049.029562:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788573049.029572:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573049.029573:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488598656/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573049.029582:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573049.029585:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573049.029586:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573049.029591:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4197 00000100:00080000:1.0:1788573049.029595:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573049.029600:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573049.029605:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573049.029618:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488598784/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573049.029627:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573049/real 1788573049] req@ffff94d4049bd880 x1875450488598272/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573049.029649:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd880 x1875450488598272/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573049.029659:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573049.029662:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573049.029665:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488598272:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788573049.029684:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488598400:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788573049.029702:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573049.029711:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573049/real 1788573049] req@ffff94d4049be300 x1875450488598400/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573049.029720:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488598400/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573049.029726:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573049.029728:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573049.029731:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488598400:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788573049.029734:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488598528:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788573049.029744:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573049.029752:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573049/real 1788573049] req@ffff94d4049bdc00 x1875450488598528/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573049.029765:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488598528/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573049.029773:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573049.029777:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573049.029780:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488598528:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788573049.029787:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488598656:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788573049.029806:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573049.029816:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573049/real 1788573049] req@ffff94d4049be680 x1875450488598656/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573049.029828:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488598656/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573049.029837:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573049.029840:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573049.029844:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488598656:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573049.029851:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488598784:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788573049.029878:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573049.029887:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573049/real 1788573049] req@ffff94d4049bca80 x1875450488598784/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573049.029895:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488598784/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573049.029900:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573049.029902:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573049.029904:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488598784:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573049.541341:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573049.989441:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573050.565345:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573051.013395:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573051.589319:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573052.037430:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573052.613269:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573053.061417:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573053.637336:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573054.085346:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573054.149379:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573054.149392:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573054.149398:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573054.149409:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4202 00000100:00080000:0.0:1788573054.149421:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573054.149429:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573054.149475:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488598912/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573054.149510:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573054.149513:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573054.149515:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573054.149518:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4202 00000100:00080000:0.0:1788573054.149521:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573054.149524:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573054.149533:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573054.149542:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488599040/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573054.149550:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573054.149554:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573054.149555:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573054.149558:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4202 00000100:00080000:0.0:1788573054.149560:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573054.149566:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573054.149569:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488598912:192.168.202.125@tcp:250:kworker.0 00000008:00100000:0.0:1788573054.149570:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573054.149582:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488599168/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573054.149589:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573054.149592:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573054.149594:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573054.149597:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4202 00000100:00080000:0.0:1788573054.149600:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573054.149603:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573054.149606:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573054.149616:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f100 x1875450488599296/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573054.149622:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573054.149625:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573054.149626:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573054.149629:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4202 00000100:00080000:0.0:1788573054.149632:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573054.149634:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573054.149637:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573054.149644:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488599424/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573054.149651:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573054.149694:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573054/real 1788573054] req@ffff94d408d3ce00 x1875450488598912/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573054.149719:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488598912/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573054.149732:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573054.149736:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573054.149739:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488598912:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788573054.149746:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488599040:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788573054.149766:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573054.149776:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573054/real 1788573054] req@ffff94d408d3ea00 x1875450488599040/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573054.149785:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488599040/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573054.149791:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573054.149794:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573054.149798:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488599040:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788573054.149829:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488599168:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788573054.149852:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573054.149865:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573054/real 1788573054] req@ffff94d408d3e680 x1875450488599168/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573054.149879:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488599168/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573054.149888:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573054.149892:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573054.149896:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488599168:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788573054.149903:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488599296:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788573054.149920:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573054.149929:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573054/real 1788573054] req@ffff94d408d3f100 x1875450488599296/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573054.149937:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f100 x1875450488599296/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573054.149947:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573054.149950:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573054.149953:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488599296:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573054.149960:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488599424:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788573054.149982:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573054.149994:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573054/real 1788573054] req@ffff94d408d3ca80 x1875450488599424/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573054.150003:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488599424/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573054.150008:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573054.150011:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573054.150013:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488599424:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573054.661272:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573055.109378:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573055.685301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573056.133367:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573056.709273:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573057.157364:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573057.733274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573058.181394:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573058.757298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573059.205366:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573059.269289:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573059.269297:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573059.269301:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573059.269309:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4207 00000100:00080000:0.0:1788573059.269318:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573059.269324:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573059.269356:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f800 x1875450488599552/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573059.269385:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573059.269389:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573059.269390:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573059.269393:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4207 00000100:00080000:0.0:1788573059.269395:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573059.269397:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573059.269405:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573059.269414:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488599680/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573059.269422:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573059.269424:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573059.269426:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573059.269431:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4207 00000100:00100000:2.0:1788573059.269431:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488599552:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788573059.269436:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573059.269441:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573059.269445:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573059.269454:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488599808/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573059.269466:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573059.269468:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573059.269470:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573059.269472:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4207 00000100:00080000:0.0:1788573059.269475:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573059.269478:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573059.269481:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573059.269489:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f100 x1875450488599936/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573059.269496:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573059.269502:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573059.269503:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573059.269508:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4207 00000800:00000100:2.0:1788573059.269512:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788573059.269513:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573059.269519:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573059.269523:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573059.269530:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488600064/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573059.269558:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573059/real 1788573059] req@ffff94d408d3f800 x1875450488599552/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573059.269580:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f800 x1875450488599552/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573059.269592:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573059.269596:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573059.269601:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488599552:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788573059.269626:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488599680:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788573059.269652:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573059.269667:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573059/real 1788573059] req@ffff94d408d3d500 x1875450488599680/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573059.269680:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488599680/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573059.269689:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573059.269693:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573059.269696:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488599680:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788573059.269703:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488599808:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788573059.269724:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573059.269735:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573059/real 1788573059] req@ffff94d408d3ca80 x1875450488599808/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573059.269750:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488599808/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573059.269756:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573059.269758:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573059.269760:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488599808:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788573059.269765:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488599936:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788573059.269780:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573059.269787:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573059/real 1788573059] req@ffff94d408d3f100 x1875450488599936/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573059.269795:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f100 x1875450488599936/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573059.269801:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573059.269803:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573059.269805:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488599936:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573059.269809:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488600064:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788573059.269818:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573059.269824:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573059/real 1788573059] req@ffff94d408d3e680 x1875450488600064/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573059.269836:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488600064/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573059.269851:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573059.269855:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573059.269858:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488600064:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573059.781314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573060.229363:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573060.805345:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573061.253440:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573061.829358:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573062.277449:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573062.853353:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573063.301481:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573063.877358:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573064.325358:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573064.389342:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573064.389351:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573064.389355:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573064.389364:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4212 00000100:00080000:3.0:1788573064.389374:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573064.389380:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573064.389412:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488600192/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573064.389441:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573064.389444:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573064.389446:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573064.389450:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4212 00000100:00080000:3.0:1788573064.389453:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573064.389456:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573064.389465:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573064.389476:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488600192:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573064.389478:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488600320/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573064.389495:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573064.389498:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573064.389501:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573064.389506:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4212 00000100:00080000:3.0:1788573064.389511:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573064.389516:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573064.389522:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573064.389535:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488600448/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573064.389543:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573064.389546:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573064.389547:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573064.389550:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4212 00000800:00000100:0.0:1788573064.389555:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573064.389555:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573064.389559:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573064.389563:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573064.389572:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488600576/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573064.389579:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573064.389582:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573064.389584:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573064.389587:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4212 00000100:00080000:3.0:1788573064.389592:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573064.389595:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573064.389612:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573064/real 1788573064] req@ffff94d408c55180 x1875450488600192/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573064.389631:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488600192/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573064.389640:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573064.389644:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573064.389648:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488600192:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573064.389668:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488600320:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573064.389688:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:3.0:1788573064.389698:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573064.389699:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573064/real 1788573064] req@ffff94d408c56a00 x1875450488600320/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573064.389709:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488600320/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573064.389714:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573064.389716:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573064.389719:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488600320:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573064.389724:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488600448:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573064.389736:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573064.389742:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573064/real 1788573064] req@ffff94d408c55880 x1875450488600448/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573064.389749:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488600448/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573064.389756:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573064.389758:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573064.389760:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488600448:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573064.389764:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488600576:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573064.389778:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573064.389785:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573064/real 1788573064] req@ffff94d408c55c00 x1875450488600576/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573064.389793:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488600576/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573064.389799:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573064.389802:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573064.389805:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488600576:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573064.389815:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488600704/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573064.389875:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488600704:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788573064.389914:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573064.389937:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573064/real 1788573064] req@ffff94d408c57800 x1875450488600704/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573064.389954:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488600704/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573064.389963:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573064.389966:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573064.389971:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488600704:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573064.901298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573065.349423:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573065.925310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573066.373407:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573066.949272:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573067.397371:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573067.973306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573068.421372:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573068.997271:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573069.445383:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573069.509332:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573069.509343:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573069.509348:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573069.509359:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4217 00000100:00080000:3.0:1788573069.509372:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573069.509381:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573069.509423:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488600832/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573069.509454:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573069.509458:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573069.509459:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573069.509462:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4217 00000100:00080000:3.0:1788573069.509464:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573069.509467:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573069.509476:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573069.509487:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488600960/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573069.509506:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788573069.509509:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488600832:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573069.509510:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573069.509516:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573069.509525:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4217 00000100:00080000:3.0:1788573069.509528:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573069.509533:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573069.509539:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573069.509557:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488601088/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573069.509567:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573069.509571:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573069.509573:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573069.509577:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4217 00000100:00080000:3.0:1788573069.509581:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573069.509585:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:2.0:1788573069.509592:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:3.0:1788573069.509592:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573069.509608:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488601216/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573069.509619:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573069.509628:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573069.509630:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788573069.509633:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573069/real 1788573069] req@ffff94d408c57100 x1875450488600832/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573069.509637:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4217 00000100:00080000:3.0:1788573069.509646:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573069.509652:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573069.509657:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488600832/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788573069.509658:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788573069.509674:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788573069.509679:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488601344/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573069.509689:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573069.509695:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488600832:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788573069.509731:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488600960:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788573069.509752:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573069.509763:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573069/real 1788573069] req@ffff94d408c54380 x1875450488600960/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573069.509772:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488600960/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573069.509778:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573069.509780:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573069.509782:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488600960:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788573069.509786:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488601088:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788573069.509797:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573069.509803:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573069/real 1788573069] req@ffff94d408c54e00 x1875450488601088/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573069.509816:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488601088/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573069.509822:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573069.509823:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573069.509825:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488601088:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788573069.509829:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488601216:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788573069.509840:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573069.509845:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573069/real 1788573069] req@ffff94d408c57b80 x1875450488601216/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573069.509864:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488601216/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573069.509869:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573069.509871:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573069.509873:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488601216:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573069.509877:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488601344:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788573069.509888:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573069.509894:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573069/real 1788573069] req@ffff94d408c56d80 x1875450488601344/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573069.509901:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488601344/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573069.509934:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573069.509937:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573069.509939:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488601344:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573070.021315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573070.469407:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573071.045291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573071.493492:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573072.069322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573072.517413:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573073.093335:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573073.541389:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573074.117259:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573074.565313:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573074.629301:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573074.629311:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573074.629315:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573074.629326:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4222 00000100:00080000:0.0:1788573074.629346:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573074.629354:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573074.629390:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488601472/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573074.629417:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573074.629421:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573074.629422:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573074.629426:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4222 00000100:00080000:0.0:1788573074.629429:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573074.629432:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573074.629441:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573074.629461:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488601600/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573074.629461:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488601472:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788573074.629477:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573074.629479:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573074.629480:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573074.629483:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4222 00000100:00080000:0.0:1788573074.629485:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573074.629487:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573074.629490:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573074.629498:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488601728/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573074.629505:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573074.629512:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573074.629513:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573074.629516:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4222 00000100:00080000:0.0:1788573074.629519:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573074.629521:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573074.629525:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573074.629533:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488601856/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573074.629540:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573074.629545:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1788573074.629545:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788573074.629549:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573074.629553:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4222 00000100:00080000:0.0:1788573074.629556:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573074.629558:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573074.629562:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573074.629568:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488601984/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573074.629596:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573074/real 1788573074] req@ffff94d408d3ea00 x1875450488601472/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573074.629617:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488601472/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573074.629628:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573074.629632:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573074.629637:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488601472:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788573074.629661:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488601600:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788573074.629687:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573074.629700:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573074/real 1788573074] req@ffff94d408d3ce00 x1875450488601600/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573074.629713:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488601600/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573074.629722:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573074.629726:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573074.629729:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488601600:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788573074.629751:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488601728:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788573074.629772:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573074.629784:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573074/real 1788573074] req@ffff94d408d3df80 x1875450488601728/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573074.629797:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488601728/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573074.629806:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573074.629809:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573074.629813:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488601728:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788573074.629819:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488601856:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788573074.629839:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573074.629851:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573074/real 1788573074] req@ffff94d408d3d180 x1875450488601856/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573074.629863:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488601856/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573074.629871:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573074.629874:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573074.629879:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488601856:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573074.629885:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488601984:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788573074.629903:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573074.629915:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573074/real 1788573074] req@ffff94d408d3c000 x1875450488601984/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573074.629927:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488601984/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573074.629935:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573074.629938:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573074.629942:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488601984:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573075.141324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573075.589331:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573076.165263:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573076.613367:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573077.189315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573077.637387:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573078.213326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573078.661436:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573079.237359:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573079.685443:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573079.749370:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573079.749382:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573079.749387:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573079.749398:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4228 00000100:00080000:2.0:1788573079.749413:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573079.749423:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573079.749476:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488602112/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573079.749510:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573079.749514:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573079.749517:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573079.749522:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4228 00000100:00080000:2.0:1788573079.749527:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573079.749532:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573079.749558:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573079.749568:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488602112:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788573079.749575:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488602240/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573079.749588:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573079.749592:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573079.749594:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573079.749597:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4228 00000100:00080000:2.0:1788573079.749600:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573079.749604:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573079.749608:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573079.749620:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488602368/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573079.749627:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573079.749630:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573079.749632:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573079.749634:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4228 00000100:00080000:2.0:1788573079.749637:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573079.749639:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573079.749644:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573079.749650:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573079.749661:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63100 x1875450488602496/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573079.749670:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573079.749672:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573079.749674:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573079.749677:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4228 00000100:00080000:2.0:1788573079.749680:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573079.749683:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573079.749686:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573079.749698:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573079/real 1788573079] req@ffff94d408c61f80 x1875450488602112/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573079.749698:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60e00 x1875450488602624/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573079.749735:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488602112/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573079.749745:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573079.749748:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573079.749753:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488602112:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573079.749777:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488602240:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573079.749795:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573079.749804:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573079/real 1788573079] req@ffff94d408c61880 x1875450488602240/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573079.749812:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488602240/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573079.749818:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573079.749820:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573079.749822:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488602240:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573079.749826:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488602368:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573079.749836:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573079.749842:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573079/real 1788573079] req@ffff94d408c63480 x1875450488602368/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573079.749850:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488602368/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573079.749856:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573079.749857:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573079.749859:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488602368:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573079.749862:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488602496:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573079.749874:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573079.749880:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573079/real 1788573079] req@ffff94d408c63100 x1875450488602496/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573079.749892:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63100 x1875450488602496/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573079.749897:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573079.749899:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573079.749901:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488602496:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573079.749904:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488602624:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573079.749914:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573079.749920:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573079/real 1788573079] req@ffff94d408c60e00 x1875450488602624/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573079.749927:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60e00 x1875450488602624/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573079.749932:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573079.749934:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573079.749935:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488602624:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573080.261315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573080.709354:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573081.285318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573081.733481:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573082.309354:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573082.757414:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573083.333379:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573083.781425:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573084.357302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573084.805388:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573084.869385:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573084.869401:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573084.869407:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573084.869430:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4233 00000100:00080000:1.0:1788573084.869446:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573084.869454:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573084.869510:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488602752/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573084.869543:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573084.869546:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573084.869548:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573084.869551:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4233 00000100:00080000:1.0:1788573084.869555:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573084.869558:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573084.869569:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573084.869581:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488602880/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573084.869589:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573084.869594:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788573084.869596:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488602752:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788573084.869597:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573084.869605:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4233 00000100:00080000:1.0:1788573084.869609:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573084.869611:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573084.869614:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573084.869624:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488603008/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573084.869631:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573084.869635:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573084.869636:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573084.869640:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4233 00000100:00080000:1.0:1788573084.869641:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573084.869644:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573084.869648:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573084.869655:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488603136/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573084.869661:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573084.869663:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573084.869665:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573084.869668:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4233 00000100:00080000:1.0:1788573084.869671:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573084.869673:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573084.869677:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573084.869684:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488603264/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573084.869740:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488602880:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573084.869777:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488603008:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573084.869796:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488603136:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573084.869812:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488603264:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573085.381371:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573085.829484:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573086.405389:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573086.853353:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573087.429415:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573087.877446:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788573087.941555:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:3.0:1788573087.941617:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788573087.941641:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:3.0:1788573087.941689:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:3.0:1788573087.941699:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:3.0:1788573087.941709:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:3.0:1788573087.941721:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:0.0:1788573087.941765:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573084/real 1788573087] req@ffff94d4049bf100 x1875450488602880/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573087.941800:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488602880/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573087.941813:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573087.941820:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573087.941825:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488602880:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573087.941836:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573084/real 1788573087] req@ffff94d4049bd500 x1875450488603008/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573087.941850:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488603008/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573087.941860:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573087.941864:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573087.941867:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488603008:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573087.941884:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573084/real 1788573087] req@ffff94d4049bf800 x1875450488603136/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573087.941904:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488603136/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573087.941919:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573087.941923:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573087.941927:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488603136:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573087.941937:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573084/real 1788573087] req@ffff94d4049bed80 x1875450488603264/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573087.941950:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488603264/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573087.941959:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573087.941962:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573087.941978:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488603264:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573087.941986:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573084/real 1788573087] req@ffff94d4049bc700 x1875450488602752/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573087.941999:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488602752/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573087.942007:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573087.942012:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573087.942015:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488602752:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788573088.453285:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573088.901464:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573089.477338:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573089.925417:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573089.989354:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573089.989365:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573089.989369:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573089.989380:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4238 00000100:00080000:2.0:1788573089.989392:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573089.989411:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573089.989461:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488603392/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573089.989504:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573089.989509:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573089.989511:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573089.989516:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4238 00000100:00080000:2.0:1788573089.989520:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573089.989525:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573089.989538:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573089.989553:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488603520/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573089.989566:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573089.989573:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573089.989580:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573089.989581:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488603392:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788573089.989589:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4238 00000100:00080000:2.0:1788573089.989599:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573089.989607:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573089.989613:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573089.989631:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488603648/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573089.989643:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573089.989648:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573089.989650:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573089.989655:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4238 00000100:00080000:2.0:1788573089.989670:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788573089.989677:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788573089.989679:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573089.989689:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573089.989711:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488603776/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573089.989724:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573089.989728:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573089.989733:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573089.989735:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573089/real 1788573089] req@ffff94d408c63800 x1875450488603392/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573089.989743:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4238 00000100:00080000:2.0:1788573089.989766:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573089.989776:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488603392/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573089.989780:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573089.989796:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788573089.989801:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573089.989810:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573089.989817:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488603904/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573089.989824:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488603392:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573089.989875:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488603520:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573089.989903:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573089.989916:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573089/real 1788573089] req@ffff94d408c62a00 x1875450488603520/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573089.989929:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488603520/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573089.989937:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573089.989940:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573089.989944:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488603520:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573089.989951:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488603648:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573089.989970:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573089.989981:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573089/real 1788573089] req@ffff94d408c61500 x1875450488603648/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573089.989993:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488603648/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573089.990006:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573089.990009:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573089.990013:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488603648:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573089.990019:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488603776:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573089.990038:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573089.990048:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573089/real 1788573089] req@ffff94d408c63b80 x1875450488603776/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573089.990059:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488603776/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573089.990066:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573089.990070:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573089.990073:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488603776:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573089.990079:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488603904:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573089.990096:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573089.990106:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573089/real 1788573089] req@ffff94d408c61c00 x1875450488603904/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573089.990117:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488603904/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573089.990124:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573089.990127:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573089.990131:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488603904:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573090.501351:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573090.949408:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573091.525333:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573091.973396:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573092.549354:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573092.997336:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573093.573378:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573094.021327:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573094.597546:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573095.045399:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573095.109309:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573095.109318:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573095.109322:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573095.109330:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4243 00000100:00080000:0.0:1788573095.109339:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573095.109345:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573095.109379:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488604032/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573095.109402:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573095.109405:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573095.109407:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573095.109410:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4243 00000100:00080000:0.0:1788573095.109412:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573095.109415:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573095.109422:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573095.109432:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488604160/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573095.109437:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488604032:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788573095.109448:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573095.109455:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573095.109456:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573095.109460:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4243 00000100:00080000:0.0:1788573095.109463:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573095.109466:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573095.109470:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573095.109478:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488604288/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573095.109485:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573095.109487:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573095.109489:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573095.109491:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4243 00000100:00080000:0.0:1788573095.109493:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573095.109496:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573095.109500:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788573095.109508:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573095.109509:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488604416/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573095.109518:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573095.109521:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573095.109522:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573095.109525:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4243 00000100:00080000:0.0:1788573095.109528:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573095.109530:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573095.109533:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573095.109540:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488604544/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573095.109544:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573095/real 1788573095] req@ffff94d408d3dc00 x1875450488604032/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573095.109581:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488604032/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573095.109590:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573095.109593:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573095.109598:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488604032:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788573095.109617:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488604160:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788573095.109633:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573095.109641:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573095/real 1788573095] req@ffff94d408d3c380 x1875450488604160/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573095.109649:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488604160/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573095.109655:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573095.109657:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573095.109659:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488604160:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788573095.109663:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488604288:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788573095.109673:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573095.109678:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573095/real 1788573095] req@ffff94d408d3c700 x1875450488604288/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573095.109686:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488604288/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573095.109695:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573095.109697:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573095.109699:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488604288:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788573095.109703:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488604416:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788573095.109713:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573095.109718:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573095/real 1788573095] req@ffff94d408d3e300 x1875450488604416/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573095.109725:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488604416/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573095.109730:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573095.109731:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573095.109733:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488604416:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573095.109736:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488604544:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788573095.109745:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573095.109761:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573095/real 1788573095] req@ffff94d408d3d880 x1875450488604544/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573095.109769:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488604544/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573095.109774:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573095.109776:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573095.109778:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488604544:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573095.621478:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573096.069397:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573096.645421:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573097.093451:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573097.669325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573098.117359:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573098.693332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573099.141384:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573099.717343:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573100.165402:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573100.229345:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573100.229356:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573100.229361:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573100.229374:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4248 00000100:00080000:3.0:1788573100.229388:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573100.229398:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573100.229448:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488604672/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573100.229480:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573100.229485:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573100.229487:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573100.229490:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4248 00000100:00080000:3.0:1788573100.229494:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573100.229499:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573100.229510:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573100.229526:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55500 x1875450488604800/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573100.229538:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488604672:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573100.229540:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573100.229547:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573100.229549:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573100.229558:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4248 00000100:00080000:3.0:1788573100.229562:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573100.229564:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573100.229568:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788573100.229630:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573100.229657:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488604928/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573100.229666:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573100.229669:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573100.229671:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573100.229676:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4248 00000100:00080000:3.0:1788573100.229680:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573100.229683:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573100/real 1788573100] req@ffff94d408c54a80 x1875450488604672/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573100.229687:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573100.229708:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488604672/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573100.229719:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573100.229723:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573100.229726:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488604672:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573100.229763:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488604800:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573100.229788:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573100.229798:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573100/real 1788573100] req@ffff94d408c55500 x1875450488604800/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573100.229808:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55500 x1875450488604800/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573100.229814:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573100.229817:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573100.229819:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488604800:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573100.229825:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488604928:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573100.229840:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573100.229848:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573100/real 1788573100] req@ffff94d408c56680 x1875450488604928/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788573100.229849:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573100.229862:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488604928/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573100.229869:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488605056/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573100.229873:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573100.229880:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573100.229884:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488604928:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788573100.229924:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573100.229927:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573100.229929:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573100.229935:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4248 00000100:00080000:3.0:1788573100.229939:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573100.229946:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488605056:192.168.202.125@tcp:8:kworker.0 00000100:00080000:3.0:1788573100.229946:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573100.229955:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573100.229969:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488605184/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573100.229975:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573100.229991:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573100/real 1788573100] req@ffff94d408c54a80 x1875450488605056/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573100.230002:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488605056/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573100.230010:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573100.230013:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573100.230015:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488605056:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573100.230028:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488605184:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573100.230048:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573100.230060:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573100/real 1788573100] req@ffff94d408c56d80 x1875450488605184/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573100.230073:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488605184/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573100.230081:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573100.230084:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573100.230087:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488605184:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573100.741327:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573101.189404:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573101.765288:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573102.213388:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573102.789277:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573103.237438:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573103.813252:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573104.261332:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573104.837263:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573105.285369:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573105.349296:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573105.349304:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573105.349307:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573105.349317:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4253 00000100:00080000:3.0:1788573105.349326:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573105.349332:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573105.349364:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488605312/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573105.349404:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573105.349409:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573105.349415:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573105.349420:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4253 00000100:00080000:3.0:1788573105.349424:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573105.349430:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573105.349438:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573105.349454:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488605440/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573105.349455:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488605312:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573105.349476:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573105.349480:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573105.349483:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573105.349487:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4253 00000100:00080000:3.0:1788573105.349493:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573105.349499:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573105.349505:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573105.349520:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488605568/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573105.349527:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573105.349539:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573105.349543:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573105.349545:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573105.349579:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4253 00000100:00100000:0.0:1788573105.349582:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573105/real 1788573105] req@ffff94d408c57b80 x1875450488605312/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573105.349587:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573105.349595:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573105.349605:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573105.349614:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488605312/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573105.349618:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488605696/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573105.349636:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573105.349645:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573105.349646:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:3.0:1788573105.349651:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573105.349654:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573105.349656:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488605312:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573105.349659:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4253 00000100:00080000:3.0:1788573105.349666:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573105.349670:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573105.349673:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573105.349680:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488605824/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573105.349688:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488605440:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573105.349715:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573105.349728:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573105/real 1788573105] req@ffff94d408c54e00 x1875450488605440/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573105.349741:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488605440/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573105.349751:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573105.349754:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573105.349758:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488605440:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573105.349765:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488605568:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573105.349780:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573105.349789:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573105/real 1788573105] req@ffff94d408c54380 x1875450488605568/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573105.349801:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488605568/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573105.349810:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573105.349813:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573105.349816:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488605568:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573105.349822:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488605696:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573105.349842:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573105.349852:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573105/real 1788573105] req@ffff94d408c57100 x1875450488605696/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573105.349865:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488605696/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573105.349874:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573105.349877:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573105.349882:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488605696:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573105.349896:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488605824:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573105.349912:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573105.349922:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573105/real 1788573105] req@ffff94d408c57800 x1875450488605824/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573105.349934:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488605824/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573105.349943:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573105.349946:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573105.349949:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488605824:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573105.861265:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573106.309340:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573106.885272:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573107.333400:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573107.909254:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573108.357364:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573108.933324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573109.381361:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573109.957308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573110.405403:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573110.469295:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573110.469304:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573110.469307:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573110.469315:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4258 00000100:00080000:3.0:1788573110.469324:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573110.469330:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573110.469379:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488605952/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573110.469403:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573110.469406:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573110.469408:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573110.469410:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4258 00000100:00080000:3.0:1788573110.469413:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573110.469416:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573110.469424:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573110.469432:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488606080/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573110.469440:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573110.469443:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573110.469444:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573110.469446:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4258 00000100:00080000:3.0:1788573110.469449:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573110.469451:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573110.469454:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573110.469463:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488606208/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573110.469467:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488605952:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573110.469472:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573110.469478:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573110.469480:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573110.469483:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4258 00000100:00080000:3.0:1788573110.469485:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573110.469488:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573110.469491:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573110.469500:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488606336/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573110.469507:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573110.469510:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573110.469511:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573110.469513:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4258 00000100:00080000:3.0:1788573110.469518:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573110.469520:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573110.469523:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573110.469529:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488606464/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573110.469545:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573110.469584:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573110/real 1788573110] req@ffff94d408c55c00 x1875450488605952/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573110.469598:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488605952/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573110.469607:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573110.469609:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573110.469612:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488605952:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573110.469619:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488606080:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573110.469637:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573110.469646:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573110/real 1788573110] req@ffff94d408c55880 x1875450488606080/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573110.469654:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488606080/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573110.469660:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573110.469662:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573110.469665:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488606080:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573110.469682:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488606208:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573110.469693:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573110.469700:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573110/real 1788573110] req@ffff94d408c56a00 x1875450488606208/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573110.469707:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488606208/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573110.469713:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573110.469715:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573110.469716:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488606208:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573110.469720:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488606336:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573110.469730:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573110.469736:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573110/real 1788573110] req@ffff94d408c55180 x1875450488606336/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573110.469744:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488606336/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573110.469750:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573110.469751:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573110.469763:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488606336:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573110.469766:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488606464:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573110.469778:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573110.469785:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573110/real 1788573110] req@ffff94d408c54000 x1875450488606464/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573110.469792:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488606464/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573110.469797:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573110.469799:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573110.469801:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488606464:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573110.981273:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573111.429332:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573112.005282:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573112.453382:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573113.029319:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573113.477346:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573114.053388:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573114.501345:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573115.077278:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573115.525364:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573115.589333:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573115.589342:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573115.589345:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573115.589354:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4263 00000100:00080000:3.0:1788573115.589363:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573115.589369:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573115.589402:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488606592/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573115.589427:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573115.589430:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573115.589431:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573115.589434:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4263 00000100:00080000:3.0:1788573115.589436:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573115.589439:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573115.589445:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573115.589459:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488606720/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573115.589467:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573115.589470:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573115.589471:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573115.589477:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4263 00000100:00100000:0.0:1788573115.589478:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488606592:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573115.589481:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573115.589488:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573115.589493:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573115.589508:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488606848/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573115.589516:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573115.589525:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573115.589527:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573115.589529:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4263 00000100:00080000:3.0:1788573115.589533:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788573115.589537:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573115.589538:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573115.589544:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573115.589554:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488606976/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573115.589564:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573115.589568:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573115.589570:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573115.589573:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573115/real 1788573115] req@ffff94d408c57480 x1875450488606592/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573115.589578:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4263 00000100:00080000:3.0:1788573115.589587:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573115.589594:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573115.589597:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488606592/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788573115.589602:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788573115.589607:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573115.589611:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573115.589616:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488606592:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573115.589626:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488607104/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573115.589639:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488606720:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573115.589660:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573115.589674:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573115/real 1788573115] req@ffff94d408c54700 x1875450488606720/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573115.589688:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488606720/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573115.589697:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573115.589701:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573115.589704:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488606720:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573115.589711:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488606848:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573115.589729:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573115.589737:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573115/real 1788573115] req@ffff94d408c55f80 x1875450488606848/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573115.589745:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488606848/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573115.589751:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573115.589753:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573115.589755:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488606848:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573115.589759:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488606976:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573115.589770:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573115.589776:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573115/real 1788573115] req@ffff94d408c56300 x1875450488606976/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573115.589783:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488606976/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573115.589789:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573115.589790:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573115.589792:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488606976:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573115.589796:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488607104:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573115.589805:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573115.589810:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573115/real 1788573115] req@ffff94d408c54000 x1875450488607104/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573115.589818:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488607104/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573115.589822:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573115.589824:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573115.589826:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488607104:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573116.101325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573116.549381:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573117.125333:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573117.573420:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573118.149311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573118.597419:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573119.173335:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573119.621358:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573120.197313:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573120.645370:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573120.709342:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573120.709355:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573120.709359:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573120.709370:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4269 00000100:00080000:3.0:1788573120.709383:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573120.709391:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573120.709435:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488607232/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573120.709463:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573120.709467:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573120.709468:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573120.709471:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4269 00000100:00080000:3.0:1788573120.709473:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573120.709476:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573120.709485:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573120.709493:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488607360/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573120.709501:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573120.709504:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573120.709505:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573120.709508:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4269 00000100:00080000:3.0:1788573120.709516:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573120.709519:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573120.709522:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573120.709531:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488607488/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573120.709540:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573120.709543:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573120.709544:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573120.709548:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4269 00000100:00080000:3.0:1788573120.709551:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573120.709557:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573120.709560:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573120.709567:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488607616/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573120.709569:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488607232:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573120.709575:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573120.709585:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573120.709586:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573120.709590:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4269 00000100:00080000:3.0:1788573120.709592:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573120.709594:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573120.709598:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573120.709606:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488607744/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573120.709666:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573120.709712:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573120/real 1788573120] req@ffff94d408c55180 x1875450488607232/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573120.709735:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488607232/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573120.709747:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573120.709752:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573120.709756:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488607232:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573120.709765:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488607360:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573120.709791:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573120.709805:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573120/real 1788573120] req@ffff94d408c56a00 x1875450488607360/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573120.709818:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488607360/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573120.709827:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573120.709831:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573120.709834:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488607360:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573120.709863:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488607488:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573120.709885:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573120.709897:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573120/real 1788573120] req@ffff94d408c55880 x1875450488607488/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573120.709916:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488607488/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573120.709925:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573120.709928:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573120.709931:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488607488:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573120.709939:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488607616:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573120.709991:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573120.710007:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573120/real 1788573120] req@ffff94d408c55c00 x1875450488607616/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573120.710021:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488607616/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573120.710029:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573120.710033:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573120.710036:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488607616:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573120.710043:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488607744:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573120.710065:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573120.710076:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573120/real 1788573120] req@ffff94d408c57800 x1875450488607744/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573120.710089:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488607744/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573120.710103:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573120.710107:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573120.710110:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488607744:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573121.221295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573121.669458:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573122.245331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573122.693399:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573123.269304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573123.717421:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573124.293369:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573124.741460:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573125.317348:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573125.765417:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573125.829299:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573125.829307:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573125.829311:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573125.829320:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4274 00000100:00080000:3.0:1788573125.829329:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573125.829335:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573125.829369:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488607872/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573125.829394:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573125.829397:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573125.829398:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573125.829401:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4274 00000100:00080000:3.0:1788573125.829403:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573125.829405:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573125.829412:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573125.829420:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488608000/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573125.829427:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573125.829430:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573125.829431:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573125.829434:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4274 00000100:00080000:3.0:1788573125.829437:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573125.829439:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573125.829442:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573125.829446:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488607872:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573125.829453:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488608128/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573125.829466:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573125.829468:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573125.829469:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573125.829472:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4274 00000100:00080000:3.0:1788573125.829474:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573125.829476:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573125.829480:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573125.829488:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488608256/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573125.829494:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573125.829496:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573125.829498:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573125.829500:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4274 00000100:00080000:3.0:1788573125.829502:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573125.829505:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573125.829508:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573125.829510:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573125.829517:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488608384/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573125.829555:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573125/real 1788573125] req@ffff94d408c57100 x1875450488607872/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573125.829570:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488607872/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573125.829578:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573125.829581:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573125.829584:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488607872:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573125.829589:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488608000:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573125.829608:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573125.829617:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573125/real 1788573125] req@ffff94d408c54380 x1875450488608000/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573125.829625:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488608000/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573125.829630:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573125.829633:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573125.829635:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488608000:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573125.829653:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488608128:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573125.829665:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573125.829671:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573125/real 1788573125] req@ffff94d408c54e00 x1875450488608128/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573125.829679:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488608128/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573125.829684:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573125.829686:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573125.829688:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488608128:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573125.829692:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488608256:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573125.829702:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573125.829708:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573125/real 1788573125] req@ffff94d408c57b80 x1875450488608256/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573125.829715:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488608256/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573125.829721:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573125.829722:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573125.829724:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488608256:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573125.829727:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488608384:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573125.829737:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573125.829743:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573125/real 1788573125] req@ffff94d408c56d80 x1875450488608384/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573125.829750:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488608384/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573125.829755:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573125.829756:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573125.829758:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488608384:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573126.341298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573126.789348:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573127.365291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573127.813331:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573128.389302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573128.837355:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573129.413286:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573129.861509:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573130.437296:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573130.885382:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573130.949361:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573130.949373:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573130.949378:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573130.949390:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4279 00000100:00080000:3.0:1788573130.949403:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573130.949411:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573130.949458:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488608512/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573130.949491:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573130.949495:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573130.949500:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573130.949503:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4279 00000100:00080000:3.0:1788573130.949506:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573130.949509:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573130.949518:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573130.949528:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488608640/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573130.949536:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573130.949539:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573130.949541:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573130.949545:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4279 00000100:00080000:3.0:1788573130.949550:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573130.949551:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488608512:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573130.949555:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573130.949562:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573130.949584:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55500 x1875450488608768/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573130.949591:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573130.949594:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573130.949595:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573130.949598:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4279 00000100:00080000:3.0:1788573130.949601:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573130.949603:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573130.949607:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573130.949618:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488608896/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573130.949625:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573130.949628:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573130.949629:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573130.949631:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4279 00000100:00080000:3.0:1788573130.949633:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573130.949636:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573130.949641:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573130.949656:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488609024/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573130.949657:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573130.949713:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573130/real 1788573130] req@ffff94d408c54a80 x1875450488608512/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573130.949732:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488608512/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573130.949741:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573130.949744:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573130.949748:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488608512:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573130.949754:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488608640:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573130.949774:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573130.949784:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573130/real 1788573130] req@ffff94d408c56680 x1875450488608640/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573130.949793:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488608640/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573130.949798:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573130.949800:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573130.949803:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488608640:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573130.949825:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488608768:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573130.949836:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573130.949843:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573130/real 1788573130] req@ffff94d408c55500 x1875450488608768/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573130.949850:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55500 x1875450488608768/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573130.949856:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573130.949858:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573130.949859:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488608768:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573130.949863:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488608896:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573130.949874:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573130.949880:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573130/real 1788573130] req@ffff94d408c56d80 x1875450488608896/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573130.949887:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488608896/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573130.949893:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573130.949895:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573130.949897:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488608896:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573130.949900:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488609024:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573130.949910:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573130.949915:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573130/real 1788573130] req@ffff94d408c57b80 x1875450488609024/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573130.949922:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488609024/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573130.949927:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573130.949929:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573130.949931:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488609024:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573131.461364:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573131.909418:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573132.485288:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573132.933370:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573133.509348:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573133.957494:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573134.533263:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573134.981446:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573135.557353:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573136.005370:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573136.069371:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573136.069417:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573136.069422:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573136.069432:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4284 00000100:00080000:0.0:1788573136.069442:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573136.069449:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573136.069484:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488609152/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573136.069516:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573136.069519:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573136.069520:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573136.069523:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4284 00000100:00080000:0.0:1788573136.069525:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573136.069528:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573136.069535:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573136.069546:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ed80 x1875450488609280/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573136.069553:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573136.069557:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573136.069559:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573136.069562:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4284 00000100:00080000:0.0:1788573136.069565:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788573136.069568:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488609152:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788573136.069569:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573136.069576:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573136.069594:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488609408/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573136.069602:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573136.069605:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573136.069608:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573136.069613:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4284 00000100:00080000:0.0:1788573136.069617:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573136.069621:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573136.069628:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573136.069637:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488609536/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573136.069644:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573136.069647:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573136.069648:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573136.069650:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4284 00000100:00080000:0.0:1788573136.069654:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573136.069658:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:3.0:1788573136.069661:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:0.0:1788573136.069663:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573136.069674:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488609664/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573136.069780:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573136/real 1788573136] req@ffff94d408d3f480 x1875450488609152/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573136.069802:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488609152/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573136.069814:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573136.069818:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573136.069823:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488609152:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573136.069831:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488609280:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573136.069998:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573136.070076:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573136/real 1788573136] req@ffff94d408d3ed80 x1875450488609280/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573136.070091:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ed80 x1875450488609280/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573136.070100:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573136.070104:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573136.070107:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488609280:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573136.070131:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488609408:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573136.070153:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573136.070166:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573136/real 1788573136] req@ffff94d408d3d880 x1875450488609408/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573136.070180:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488609408/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573136.070190:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573136.070193:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573136.070196:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488609408:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573136.070236:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488609536:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573136.070261:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573136.070282:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573136/real 1788573136] req@ffff94d408d3e300 x1875450488609536/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573136.070297:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488609536/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573136.070306:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573136.070310:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573136.070313:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488609536:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573136.070321:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488609664:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573136.070342:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573136.070353:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573136/real 1788573136] req@ffff94d408d3c700 x1875450488609664/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573136.070365:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488609664/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573136.070398:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573136.070402:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573136.070406:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488609664:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573136.581295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573137.029371:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573137.605381:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573138.053486:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573138.629285:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573139.077418:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573139.653316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573140.101472:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573140.677328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573141.125430:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573141.189314:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573141.189323:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573141.189327:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573141.189335:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4289 00000100:00080000:3.0:1788573141.189344:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573141.189350:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573141.189381:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488609792/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573141.189407:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573141.189411:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573141.189413:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573141.189417:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4289 00000100:00080000:3.0:1788573141.189421:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573141.189426:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573141.189436:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573141.189452:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488609920/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573141.189457:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488609792:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573141.189473:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573141.189475:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573141.189477:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573141.189480:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4289 00000100:00080000:3.0:1788573141.189483:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573141.189486:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573141.189490:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573141.189501:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488610048/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573141.189508:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573141.189510:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573141.189511:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573141.189514:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4289 00000100:00080000:3.0:1788573141.189523:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573141.189526:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573141.189529:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573141.189534:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573141.189538:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488610176/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573141.189546:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573141.189548:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573141.189549:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573141.189552:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4289 00000100:00080000:3.0:1788573141.189556:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573141.189558:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573141.189561:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573141.189568:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488610304/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573141.189583:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573141/real 1788573141] req@ffff94d408c54e00 x1875450488609792/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573141.189605:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488609792/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573141.189625:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573141.189629:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573141.189634:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488609792:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573141.189659:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488609920:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573141.189697:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573141.189712:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573141/real 1788573141] req@ffff94d408c54380 x1875450488609920/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573141.189727:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488609920/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573141.189736:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573141.189740:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573141.189743:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488609920:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573141.189751:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488610048:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573141.189772:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573141.189784:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573141/real 1788573141] req@ffff94d408c57100 x1875450488610048/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573141.189798:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488610048/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573141.189807:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573141.189810:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573141.189814:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488610048:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573141.189821:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488610176:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573141.189842:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573141.189855:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573141/real 1788573141] req@ffff94d408c57800 x1875450488610176/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573141.189867:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488610176/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573141.189877:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573141.189881:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573141.189884:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488610176:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573141.189891:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488610304:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573141.189911:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573141.189923:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573141/real 1788573141] req@ffff94d408c55c00 x1875450488610304/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573141.189937:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488610304/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573141.189946:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573141.189950:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573141.189954:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488610304:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573141.701307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573142.149374:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573142.725283:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573143.173349:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573143.749353:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573144.197455:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573144.773332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573145.221344:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573145.797334:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573146.245431:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573146.309394:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573146.309408:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573146.309414:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573146.309428:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4294 00000100:00080000:3.0:1788573146.309443:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573146.309455:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573146.309519:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488610432/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573146.309557:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573146.309562:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573146.309564:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573146.309569:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4294 00000100:00080000:3.0:1788573146.309573:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573146.309579:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573146.309592:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573146.309610:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488610560/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573146.309618:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488610432:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573146.309628:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573146.309634:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573146.309637:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573146.309642:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4294 00000100:00080000:3.0:1788573146.309646:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573146.309651:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573146.309656:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573146.309671:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488610688/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573146.309684:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573146.309688:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573146.309690:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573146.309694:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4294 00000100:00080000:3.0:1788573146.309708:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573146.309712:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573146.309717:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573146.309721:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573146.309736:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488610816/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573146.309750:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573146.309754:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573146.309757:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573146.309764:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4294 00000100:00080000:3.0:1788573146.309769:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573146.309771:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573146/real 1788573146] req@ffff94d408c55880 x1875450488610432/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573146.309782:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573146.309792:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573146.309795:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488610432/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573146.309807:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573146.309812:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573146.309813:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488610944/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573146.309819:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488610432:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573146.309849:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488610560:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573146.309870:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573146.309881:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573146/real 1788573146] req@ffff94d408c56a00 x1875450488610560/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573146.309890:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488610560/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573146.309896:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573146.309899:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573146.309901:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488610560:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573146.309904:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488610688:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573146.309915:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573146.309921:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573146/real 1788573146] req@ffff94d408c55180 x1875450488610688/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573146.309934:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488610688/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573146.309939:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573146.309941:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573146.309943:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488610688:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573146.309946:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488610816:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573146.309958:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573146.309964:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573146/real 1788573146] req@ffff94d408c54000 x1875450488610816/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573146.309972:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488610816/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573146.309977:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573146.309978:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573146.309980:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488610816:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573146.309984:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488610944:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573146.309993:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573146.309999:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573146/real 1788573146] req@ffff94d408c56300 x1875450488610944/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573146.310007:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488610944/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573146.310016:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573146.310040:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573146.310043:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488610944:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573146.821323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573147.269339:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573147.845305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573148.293379:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573148.869292:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573149.317330:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573149.893306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573150.341352:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573150.917316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573151.365359:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573151.429293:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573151.429303:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573151.429306:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573151.429315:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4299 00000100:00080000:2.0:1788573151.429324:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573151.429330:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573151.429363:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488611072/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573151.429387:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573151.429390:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573151.429391:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573151.429394:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4299 00000100:00080000:2.0:1788573151.429396:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573151.429398:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573151.429405:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573151.429412:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488611200/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573151.429420:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573151.429422:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573151.429424:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573151.429427:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4299 00000100:00080000:2.0:1788573151.429429:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573151.429432:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573151.429435:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573151.429445:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488611328/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573151.429446:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488611072:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788573151.429458:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573151.429460:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573151.429462:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573151.429464:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4299 00000100:00080000:2.0:1788573151.429466:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573151.429469:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573151.429472:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573151.429479:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488611456/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573151.429485:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573151.429487:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573151.429489:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573151.429491:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4299 00000100:00080000:2.0:1788573151.429494:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573151.429496:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573151.429498:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573151.429504:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488611584/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573151.429542:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488611200:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573151.429580:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488611328:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573151.429592:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488611456:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573151.429607:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488611584:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573151.941282:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573152.389307:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573152.965299:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573153.413388:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573153.989304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573154.437394:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788573154.501496:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:3.0:1788573154.501540:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788573154.501557:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:3.0:1788573154.501596:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:3.0:1788573154.501603:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:3.0:1788573154.501610:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:3.0:1788573154.501617:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:0.0:1788573154.501714:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573151/real 1788573154] req@ffff94d408c62d80 x1875450488611328/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573154.501804:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488611328/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573154.501823:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573154.501830:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573154.501836:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488611328:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573154.501848:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573151/real 1788573154] req@ffff94d408c60000 x1875450488611456/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573154.501868:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488611456/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573154.501880:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573154.501885:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573154.501890:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488611456:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573154.501898:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573151/real 1788573154] req@ffff94d408c60a80 x1875450488611584/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573154.501915:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488611584/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573154.501926:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573154.501931:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573154.501936:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488611584:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573154.501945:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573151/real 1788573154] req@ffff94d408c62680 x1875450488611072/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573154.501962:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488611072/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573154.501972:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573154.501976:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573154.501980:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488611072:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573154.501988:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573151/real 1788573154] req@ffff94d408c61180 x1875450488611200/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573154.502004:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488611200/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573154.502013:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573154.502017:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573154.502021:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488611200:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788573155.013310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573155.461413:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573156.037273:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573156.485366:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573156.549339:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573156.549348:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573156.549351:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573156.549359:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4304 00000100:00080000:1.0:1788573156.549368:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573156.549376:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573156.549407:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488611712/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573156.549436:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573156.549440:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573156.549453:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573156.549457:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4304 00000100:00080000:1.0:1788573156.549461:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573156.549466:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573156.549474:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573156.549486:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488611840/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573156.549498:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573156.549501:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573156.549502:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573156.549504:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488611712:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788573156.549506:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4304 00000100:00080000:1.0:1788573156.549512:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573156.549516:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573156.549520:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573156.549529:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdf80 x1875450488611968/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573156.549536:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573156.549538:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573156.549539:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573156.549542:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4304 00000100:00080000:1.0:1788573156.549544:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573156.549547:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573156.549555:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573156.549563:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488612096/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573156.549573:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788573156.549573:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573156.549580:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573156.549583:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573156.549588:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4304 00000100:00080000:1.0:1788573156.549592:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573156.549597:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573156.549602:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573156.549614:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573156/real 1788573156] req@ffff94d4049bc000 x1875450488611712/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573156.549630:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488612224/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573156.549637:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488611712/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573156.549659:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573156.549677:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573156.549683:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488611712:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573156.549708:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488611840:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573156.549735:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573156.549748:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573156/real 1788573156] req@ffff94d4049bfb80 x1875450488611840/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573156.549777:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488611840/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573156.549795:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573156.549800:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573156.549803:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488611840:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573156.549809:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488611968:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573156.549829:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573156.549840:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573156/real 1788573156] req@ffff94d4049bdf80 x1875450488611968/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573156.549852:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdf80 x1875450488611968/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573156.549860:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573156.549863:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573156.549866:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488611968:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573156.549872:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488612096:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573156.549890:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573156.549901:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573156/real 1788573156] req@ffff94d4049bea00 x1875450488612096/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573156.549914:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488612096/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573156.549923:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573156.549926:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573156.549929:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488612096:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573156.549933:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488612224:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573156.549953:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573156.549964:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573156/real 1788573156] req@ffff94d4049bf480 x1875450488612224/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573156.549976:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488612224/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573156.549985:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573156.549988:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573156.549991:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488612224:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573157.061316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573157.509411:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573158.085296:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573158.533428:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573159.109402:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573159.557408:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573160.134266:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573160.581455:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573161.158066:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573161.605382:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573161.669350:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573161.669358:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573161.669361:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573161.669369:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4309 00000100:00080000:3.0:1788573161.669379:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573161.669385:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573161.669417:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488612352/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573161.669441:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573161.669443:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573161.669445:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573161.669448:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4309 00000100:00080000:3.0:1788573161.669450:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573161.669453:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573161.669460:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573161.669468:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488612480/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573161.669476:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573161.669478:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573161.669480:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573161.669496:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4309 00000100:00080000:3.0:1788573161.669500:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573161.669504:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488612352:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573161.669506:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573161.669514:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573161.669527:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488612608/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573161.669535:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573161.669538:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573161.669539:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573161.669542:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4309 00000100:00080000:3.0:1788573161.669546:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573161.669549:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573161.669553:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573161.669563:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488612736/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573161.669567:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573161.669588:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573161.669593:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573161.669595:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573161.669600:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4309 00000100:00100000:0.0:1788573161.669605:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573161/real 1788573161] req@ffff94d408c55f80 x1875450488612352/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573161.669607:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573161.669616:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573161.669625:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573161.669636:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488612352/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573161.669640:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488612864/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573161.669672:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573161.669677:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573161.669680:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488612352:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573161.669689:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488612480:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573161.669709:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573161.669718:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573161/real 1788573161] req@ffff94d408c54700 x1875450488612480/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573161.669727:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488612480/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573161.669732:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573161.669735:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573161.669737:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488612480:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573161.669756:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488612608:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573161.669766:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573161.669773:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573161/real 1788573161] req@ffff94d408c57480 x1875450488612608/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573161.669780:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488612608/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573161.669786:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573161.669788:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573161.669789:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488612608:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573161.669793:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488612736:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573161.669803:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573161.669808:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573161/real 1788573161] req@ffff94d408c56300 x1875450488612736/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573161.669816:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488612736/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573161.669820:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573161.669822:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573161.669824:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488612736:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573161.669827:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488612864:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573161.669836:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573161.669841:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573161/real 1788573161] req@ffff94d408c54000 x1875450488612864/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573161.669847:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488612864/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573161.669852:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573161.669854:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573161.669855:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488612864:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573162.181730:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573162.629356:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573163.206682:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573163.653339:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573164.229774:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573164.677421:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573165.253501:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573165.701405:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573166.277304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573166.725392:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573166.789355:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573166.789365:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573166.789370:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573166.789381:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4315 00000100:00080000:3.0:1788573166.789395:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573166.789403:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573166.789448:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488612992/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573166.789476:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573166.789480:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573166.789481:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573166.789484:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4315 00000100:00080000:3.0:1788573166.789487:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573166.789489:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573166.789505:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573166.789515:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488613120/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573166.789524:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573166.789529:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788573166.789531:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488612992:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573166.789531:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573166.789540:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4315 00000100:00080000:3.0:1788573166.789544:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573166.789546:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573166.789549:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573166.789560:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488613248/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573166.789567:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573166.789570:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573166.789571:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573166.789574:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4315 00000100:00080000:3.0:1788573166.789577:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573166.789581:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573166.789585:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573166.789591:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488613376/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573166.789598:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573166.789600:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573166.789602:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573166.789604:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4315 00000100:00080000:3.0:1788573166.789607:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573166.789612:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:0.0:1788573166.789614:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:3.0:1788573166.789616:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573166.789625:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488613504/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573166.789658:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573166/real 1788573166] req@ffff94d408c55180 x1875450488612992/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573166.789677:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488612992/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573166.789687:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573166.789690:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573166.789694:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488612992:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573166.789717:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488613120:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573166.789734:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573166.789743:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573166/real 1788573166] req@ffff94d408c56a00 x1875450488613120/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573166.789752:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488613120/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573166.789758:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573166.789761:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573166.789763:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488613120:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573166.789766:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488613248:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573166.789777:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573166.789782:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573166/real 1788573166] req@ffff94d408c55880 x1875450488613248/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573166.789802:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488613248/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573166.789809:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573166.789811:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573166.789813:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488613248:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573166.789818:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488613376:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573166.789832:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573166.789838:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573166/real 1788573166] req@ffff94d408c55c00 x1875450488613376/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573166.789846:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488613376/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573166.789852:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573166.789854:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573166.789856:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488613376:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573166.789860:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488613504:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573166.789871:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573166.789882:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573166/real 1788573166] req@ffff94d408c57800 x1875450488613504/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573166.789893:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488613504/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573166.789902:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573166.789906:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573166.789910:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488613504:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573167.301359:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573167.749414:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573168.325328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573168.773425:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573169.349321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573169.797383:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573170.373300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573170.821399:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573171.397294:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573171.845298:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573171.909280:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573171.909292:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573171.909296:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573171.909308:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4320 00000100:00080000:3.0:1788573171.909317:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573171.909324:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573171.909357:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488613632/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573171.909381:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573171.909384:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573171.909387:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573171.909391:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4320 00000100:00080000:3.0:1788573171.909395:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573171.909399:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573171.909407:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573171.909411:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488613632:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573171.909423:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488613760/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573171.909433:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573171.909436:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573171.909439:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573171.909444:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4320 00000100:00080000:3.0:1788573171.909449:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573171.909453:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573171.909459:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573171.909477:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488613888/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573171.909489:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573171.909494:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573171.909497:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573171.909503:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4320 00000100:00080000:3.0:1788573171.909510:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788573171.909517:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573171.909524:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573171.909535:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573171.909582:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573171/real 1788573171] req@ffff94d408c57100 x1875450488613632/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573171.909610:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488614016/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573171.909616:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488613632/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573171.909649:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573171.909654:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573171.909658:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488613632:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573171.909693:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488613760:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573171.909731:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573171.909749:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573171.909752:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1788573171.909754:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573171/real 1788573171] req@ffff94d408c54380 x1875450488613760/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573171.909755:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573171.909767:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4320 00000100:00100000:1.0:1788573171.909790:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488613760/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573171.909803:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573171.909806:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573171.909811:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488613760:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573171.909817:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488613888:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573171.909836:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573171.909856:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573171/real 1788573171] req@ffff94d408c54e00 x1875450488613888/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573171.909858:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573171.909873:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573171.909877:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488613888/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788573171.909889:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788573171.909892:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573171.909896:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573171.909900:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488613888:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573171.909909:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488614144/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573171.909918:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488614016:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573171.909935:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573171.909941:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573171/real 1788573171] req@ffff94d408c57b80 x1875450488614016/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573171.909949:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488614016/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573171.909955:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573171.909958:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573171.909960:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488614016:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573171.909966:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488614144:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573171.909977:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573171.909983:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573171/real 1788573171] req@ffff94d408c56d80 x1875450488614144/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573171.909991:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488614144/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573171.909996:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573171.909998:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573171.910000:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488614144:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573172.421325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573172.869375:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573173.445263:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573173.893413:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573174.469305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573174.917380:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573175.493247:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573175.941376:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573176.517311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573176.965393:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573177.029313:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573177.029322:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573177.029325:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573177.029334:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4325 00000100:00080000:0.0:1788573177.029343:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573177.029349:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573177.029381:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488614272/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573177.029403:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573177.029406:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573177.029407:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573177.029410:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4325 00000100:00080000:0.0:1788573177.029413:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573177.029415:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573177.029422:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573177.029430:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488614400/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573177.029438:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573177.029441:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573177.029443:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573177.029447:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4325 00000100:00080000:0.0:1788573177.029451:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573177.029455:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573177.029461:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573177.029464:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488614272:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573177.029474:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488614528/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573177.029488:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573177.029492:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573177.029494:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573177.029497:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4325 00000100:00080000:0.0:1788573177.029501:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573177.029505:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573177.029509:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573177.029518:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488614656/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573177.029525:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573177.029527:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573177.029528:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573177.029530:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4325 00000100:00080000:0.0:1788573177.029533:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788573177.029538:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788573177.029539:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573177.029545:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573177.029552:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488614784/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573177.029590:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573177/real 1788573177] req@ffff94d408d3c380 x1875450488614272/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573177.029612:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488614272/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573177.029624:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573177.029628:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573177.029633:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488614272:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573177.029641:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488614400:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573177.029667:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573177.029682:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573177/real 1788573177] req@ffff94d408d3dc00 x1875450488614400/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573177.029696:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488614400/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573177.029712:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573177.029716:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573177.029720:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488614400:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573177.029744:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488614528:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573177.029766:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573177.029779:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573177/real 1788573177] req@ffff94d408d3c000 x1875450488614528/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573177.029791:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488614528/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573177.029800:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573177.029804:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573177.029807:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488614528:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573177.029814:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488614656:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573177.029833:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573177.029844:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573177/real 1788573177] req@ffff94d408d3d180 x1875450488614656/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573177.029856:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488614656/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573177.029864:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573177.029867:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573177.029871:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488614656:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573177.029878:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488614784:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573177.029898:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573177.029909:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573177/real 1788573177] req@ffff94d408d3df80 x1875450488614784/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573177.029922:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488614784/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573177.029930:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573177.029934:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573177.029937:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488614784:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573177.541340:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573177.989374:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573178.565313:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573179.013332:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573179.589361:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573180.037484:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573180.613341:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573181.061391:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573181.637304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573182.085324:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573182.149339:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573182.149347:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573182.149350:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573182.149359:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4330 00000100:00080000:2.0:1788573182.149368:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573182.149375:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573182.149407:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60700 x1875450488614912/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573182.149434:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573182.149437:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573182.149438:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573182.149441:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4330 00000100:00080000:2.0:1788573182.149443:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573182.149445:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573182.149452:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573182.149460:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60380 x1875450488615040/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573182.149467:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573182.149470:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573182.149471:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573182.149473:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4330 00000100:00080000:2.0:1788573182.149475:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573182.149478:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573182.149480:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573182.149487:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62300 x1875450488615168/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573182.149492:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488614912:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788573182.149496:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573182.149502:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573182.149504:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573182.149507:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4330 00000100:00080000:2.0:1788573182.149509:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573182.149511:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573182.149514:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573182.149523:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488615296/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573182.149529:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573182.149531:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573182.149532:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573182.149535:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4330 00000100:00080000:2.0:1788573182.149537:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573182.149540:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573182.149543:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573182.149549:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488615424/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573182.149565:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573182.149595:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573182/real 1788573182] req@ffff94d408c60700 x1875450488614912/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573182.149612:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60700 x1875450488614912/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573182.149623:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573182.149627:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573182.149632:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488614912:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573182.149640:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488615040:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573182.149664:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573182.149685:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573182/real 1788573182] req@ffff94d408c60380 x1875450488615040/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573182.149700:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60380 x1875450488615040/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573182.149709:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573182.149713:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573182.149716:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488615040:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573182.149741:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488615168:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573182.149764:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573182.149789:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573182/real 1788573182] req@ffff94d408c62300 x1875450488615168/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573182.149798:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62300 x1875450488615168/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573182.149804:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573182.149806:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573182.149809:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488615168:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573182.149823:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488615296:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573182.149845:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573182.149857:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573182/real 1788573182] req@ffff94d408c61180 x1875450488615296/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573182.149868:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488615296/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573182.149874:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573182.149876:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573182.149879:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488615296:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573182.149883:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488615424:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573182.149897:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573182.149903:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573182/real 1788573182] req@ffff94d408c62680 x1875450488615424/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573182.149910:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488615424/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573182.149916:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573182.149918:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573182.149920:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488615424:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573182.661270:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573183.109417:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573183.685290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573184.133393:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573184.709345:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573185.157381:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573185.733327:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573186.181380:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573186.757327:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573187.205408:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573187.269355:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573187.269367:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573187.269371:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573187.269383:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4335 00000100:00080000:2.0:1788573187.269395:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573187.269403:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573187.269448:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488615552/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573187.269480:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573187.269484:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573187.269486:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573187.269490:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4335 00000100:00080000:2.0:1788573187.269494:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573187.269499:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573187.269510:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573187.269526:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488615680/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573187.269535:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488615552:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788573187.269540:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573187.269544:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573187.269545:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573187.269549:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4335 00000100:00080000:2.0:1788573187.269552:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573187.269555:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573187.269559:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573187.269571:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488615808/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573187.269579:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573187.269604:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573187.269607:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573187.269612:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4335 00000100:00080000:2.0:1788573187.269617:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573187.269622:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573187.269629:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573187.269630:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573187.269647:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488615936/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573187.269659:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573187.269663:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573187.269665:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573187.269669:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4335 00000100:00080000:2.0:1788573187.269674:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573187.269675:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573187/real 1788573187] req@ffff94d408c60a80 x1875450488615552/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573187.269683:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573187.269692:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573187.269698:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488615552/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573187.269709:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488616064/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573187.269710:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573187.269721:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573187.269729:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488615552:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573187.269763:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488615680:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573187.269785:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573187.269795:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573187/real 1788573187] req@ffff94d408c60000 x1875450488615680/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573187.269804:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488615680/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573187.269810:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573187.269813:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573187.269815:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488615680:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573187.269819:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488615808:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573187.269830:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573187.269836:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573187/real 1788573187] req@ffff94d408c62d80 x1875450488615808/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573187.269844:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488615808/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573187.269850:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573187.269852:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573187.269854:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488615808:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573187.269858:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488615936:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573187.269869:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573187.269875:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573187/real 1788573187] req@ffff94d408c61c00 x1875450488615936/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573187.269884:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488615936/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573187.269889:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573187.269891:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573187.269893:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488615936:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573187.269896:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488616064:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573187.269908:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573187.269914:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573187/real 1788573187] req@ffff94d408c63b80 x1875450488616064/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573187.269921:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488616064/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573187.269927:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573187.269929:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573187.269930:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488616064:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573187.781331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573188.229406:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573188.805313:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573189.253319:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573189.829310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573190.277372:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573190.853277:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573191.301348:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573191.877328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573192.325407:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573192.389343:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573192.389351:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573192.389354:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573192.389362:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4340 00000100:00080000:1.0:1788573192.389372:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573192.389378:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573192.389432:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488616192/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573192.389460:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573192.389463:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573192.389464:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573192.389467:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4340 00000100:00080000:1.0:1788573192.389469:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573192.389472:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573192.389479:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573192.389487:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488616320/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573192.389495:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573192.389498:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573192.389499:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573192.389502:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4340 00000100:00080000:1.0:1788573192.389504:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573192.389507:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573192.389509:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573192.389515:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488616448/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573192.389521:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488616192:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788573192.389524:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573192.389535:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573192.389536:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573192.389540:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4340 00000100:00080000:1.0:1788573192.389543:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573192.389546:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573192.389550:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573192.389560:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488616576/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573192.389566:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573192.389568:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573192.389570:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573192.389572:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4340 00000100:00080000:1.0:1788573192.389575:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573192.389577:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573192.389580:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573192.389587:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573192.389589:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488616704/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573192.389628:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573192/real 1788573192] req@ffff94d4049bce00 x1875450488616192/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573192.389642:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488616192/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573192.389650:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573192.389653:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573192.389655:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488616192:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573192.389661:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488616320:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573192.389685:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573192.389693:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573192/real 1788573192] req@ffff94d4049bd180 x1875450488616320/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573192.389701:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488616320/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573192.389707:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573192.389709:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573192.389711:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488616320:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573192.389733:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488616448:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573192.389754:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573192.389766:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573192/real 1788573192] req@ffff94d4049bc380 x1875450488616448/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573192.389779:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc380 x1875450488616448/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573192.389784:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573192.389786:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573192.389788:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488616448:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573192.389793:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488616576:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573192.389807:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573192.389814:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573192/real 1788573192] req@ffff94d4049bf480 x1875450488616576/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573192.389821:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488616576/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573192.389827:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573192.389829:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573192.389831:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488616576:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573192.389834:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488616704:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573192.389843:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573192.389849:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573192/real 1788573192] req@ffff94d4049bea00 x1875450488616704/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573192.389856:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488616704/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573192.389861:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573192.389863:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573192.389864:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488616704:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573192.901284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573193.349336:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573193.925315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573194.373424:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573194.949310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573195.397446:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573195.973333:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573196.421395:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573196.997349:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573197.445395:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573197.509421:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573197.509429:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573197.509433:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573197.509441:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4345 00000100:00080000:0.0:1788573197.509451:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573197.509457:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573197.509492:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488616832/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573197.509522:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573197.509525:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573197.509526:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573197.509529:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4345 00000100:00080000:0.0:1788573197.509531:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573197.509534:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573197.509541:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573197.509554:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488616960/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573197.509562:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573197.509565:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573197.509566:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573197.509568:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4345 00000100:00080000:0.0:1788573197.509570:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573197.509572:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573197.509575:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573197.509581:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488617088/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573197.509589:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573197.509592:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573197.509595:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573197.509595:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488616832:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788573197.509604:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4345 00000100:00080000:0.0:1788573197.509612:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573197.509617:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573197.509620:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573197.509628:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f100 x1875450488617216/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573197.509634:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573197.509636:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573197.509637:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573197.509641:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4345 00000100:00080000:0.0:1788573197.509644:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573197.509646:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573197.509649:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573197.509655:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488617344/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573197.509710:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573197.509764:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573197/real 1788573197] req@ffff94d408d3ce00 x1875450488616832/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573197.509790:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488616832/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573197.509804:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573197.509808:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573197.509813:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488616832:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573197.509824:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488616960:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573197.509852:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573197.509868:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573197/real 1788573197] req@ffff94d408d3ea00 x1875450488616960/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573197.509883:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488616960/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573197.509892:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573197.509896:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573197.509900:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488616960:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573197.509942:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488617088:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573197.509968:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573197.509982:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573197/real 1788573197] req@ffff94d408d3e680 x1875450488617088/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573197.509996:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488617088/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573197.510005:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573197.510009:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573197.510013:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488617088:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573197.510020:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488617216:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573197.510044:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573197.510057:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573197/real 1788573197] req@ffff94d408d3f100 x1875450488617216/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573197.510071:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f100 x1875450488617216/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573197.510080:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573197.510084:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573197.510087:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488617216:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573197.510095:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488617344:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573197.510118:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573197.510131:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573197/real 1788573197] req@ffff94d408d3ca80 x1875450488617344/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573197.510145:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488617344/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573197.510154:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573197.510157:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573197.510161:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488617344:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573198.021269:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573198.469374:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573199.045301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573199.493386:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573200.069294:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573200.517370:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573201.093291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:2.0:1788573201.541324:0:68734:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:2.0:1788573201.541344:0:68734:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 2/3 locks to regular blp list, 2 blwis in pool 00010000:00010000:3.0:1788573201.541356:0:70972:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 2 total in blp. (1 blwis in pool) 00010000:00010000:3.0:1788573201.541365:0:70972:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d408cf1e00/0x1e260726fbbc0c50 lrc: 2/0,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4e1036 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788573201.541385:0:70973:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 2 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788573201.541419:0:70972:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d408cf1e00/0x1e260726fbbc0c50 lrc: 1/0,0 mode: --/PR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4e1036 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788573201.541426:0:70972:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788573201.541451:0:70972:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff94d402e39800 lock: ffff94d408cf1e00/0x1e260726fbbc0c50 lrc: 0/0,0 mode: --/PR res: [0x240006991:0xc:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4e1036 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788573201.541469:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788573201.541471:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788573201.541473:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573201.541494:0:2405:0:(client.c:1832:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff94d408c55500 x1875450488617472/t0(0) o103->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:0.0:1788573201.541657:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573201.541660:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573201.541661:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788573201.541686:0:70973:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412166800/0x1e260726fbbc0c5e lrc: 2/0,0 mode: PR/PR res: [0x240006991:0xd:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4e1060 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788573201.541713:0:70973:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412167200/0x1e260726fbbc0c57 lrc: 2/0,0 mode: PR/PR res: [0x240006991:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xec901abfeb4e1044 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788573201.541735:0:70973:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412166800/0x1e260726fbbc0c5e lrc: 1/0,0 mode: --/PR res: [0x240006991:0xd:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4e1060 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788573201.541751:0:70973:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412167200/0x1e260726fbbc0c57 lrc: 1/0,0 mode: --/PR res: [0x240006991:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4e1044 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788573201.541758:0:70973:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1788573201.541781:0:70973:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412166800/0x1e260726fbbc0c5e lrc: 0/0,0 mode: --/PR res: [0x240006991:0xd:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4e1060 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788573201.541801:0:70973:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff94d402e39800 lock: ffff94d412167200/0x1e260726fbbc0c57 lrc: 0/0,0 mode: --/PR res: [0x240006991:0xc:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xec901abfeb4e1044 expref: -99 pid: 71063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00080000:0.0:1788573201.541804:0:2404:0:(client.c:1832:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff94d4049bdf80 x1875450488617600/t0(0) o103->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1788573201.541828:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788573201.541832:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788573201.541835:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573202.117271:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573202.565326:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573202.629309:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573202.629317:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573202.629320:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573202.629328:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4350 00000100:00080000:1.0:1788573202.629337:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573202.629343:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573202.629376:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488617728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573202.629400:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573202.629403:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573202.629404:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573202.629407:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4350 00000100:00080000:1.0:1788573202.629410:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573202.629413:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573202.629421:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573202.629430:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488617856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573202.629439:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573202.629444:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488617728:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788573202.629445:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573202.629449:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573202.629455:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4350 00000100:00080000:1.0:1788573202.629457:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573202.629460:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573202.629463:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573202.629472:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488617984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573202.629479:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573202.629482:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573202.629483:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573202.629486:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4350 00000100:00080000:1.0:1788573202.629495:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573202.629498:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573202.629502:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573202.629508:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573202.629512:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488618112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573202.629524:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573202.629527:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573202.629528:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573202.629531:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4350 00000100:00080000:1.0:1788573202.629534:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573202.629536:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573202.629540:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573202.629547:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573202/real 1788573202] req@ffff94d4049bfb80 x1875450488617728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573202.629549:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488618240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573202.629596:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488617728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573202.629609:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573202.629614:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573202.629618:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488617728:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573202.629642:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488617856:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573202.629667:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573202.629681:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573202/real 1788573202] req@ffff94d4049bc000 x1875450488617856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573202.629695:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488617856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573202.629704:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573202.629708:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573202.629711:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488617856:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573202.629718:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488617984:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573202.629740:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573202.629752:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573202/real 1788573202] req@ffff94d4049bc700 x1875450488617984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573202.629765:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488617984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573202.629774:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573202.629777:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573202.629781:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488617984:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573202.629788:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488618112:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573202.629808:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573202.629820:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573202/real 1788573202] req@ffff94d4049bed80 x1875450488618112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573202.629840:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488618112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573202.629849:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573202.629852:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573202.629856:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488618112:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573202.629862:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488618240:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573202.629882:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573202.629893:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573202/real 1788573202] req@ffff94d4049bf800 x1875450488618240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573202.629905:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488618240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573202.629914:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573202.629917:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573202.629920:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488618240:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573203.141374:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573203.589411:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573204.165309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573204.613320:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573205.189297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573205.637363:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573206.213311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573206.661376:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573207.237277:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573207.685361:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573207.749322:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573207.749334:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573207.749340:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573207.749352:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4356 00000100:00080000:1.0:1788573207.749366:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573207.749376:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573207.749421:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488618368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573207.749460:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573207.749463:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573207.749466:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573207.749471:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4356 00000100:00080000:1.0:1788573207.749477:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573207.749481:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573207.749493:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573207.749509:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488618496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573207.749519:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573207.749522:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573207.749525:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573207.749529:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4356 00000100:00080000:1.0:1788573207.749532:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573207.749537:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573207.749545:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573207.749545:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488618368:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573207.749561:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488618624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573207.749570:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573207.749573:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573207.749574:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573207.749579:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4356 00000100:00080000:1.0:1788573207.749585:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573207.749589:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573207.749593:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573207.749607:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488618752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573207.749620:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573207.749623:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573207.749625:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573207.749630:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4356 00000100:00080000:1.0:1788573207.749633:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573207.749637:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573207.749642:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573207.749648:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488618880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573207.749863:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573207.749966:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573207/real 1788573207] req@ffff94d4049bd500 x1875450488618368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573207.749985:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488618368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573207.749994:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573207.749997:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573207.750013:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488618368:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573207.750023:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488618496:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573207.750050:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573207.750061:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573207/real 1788573207] req@ffff94d4049bf100 x1875450488618496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573207.750070:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488618496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573207.750077:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573207.750079:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573207.750082:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488618496:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573207.750104:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488618624:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573207.750118:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573207.750125:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573207/real 1788573207] req@ffff94d4049bca80 x1875450488618624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573207.750140:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488618624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573207.750147:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573207.750150:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573207.750152:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488618624:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573207.750155:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488618752:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573207.750175:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573207.750187:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573207/real 1788573207] req@ffff94d4049be680 x1875450488618752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573207.750240:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488618752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573207.750248:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573207.750250:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573207.750257:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488618752:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573207.750269:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488618880:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573207.750289:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573207.750298:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573207/real 1788573207] req@ffff94d4049bdc00 x1875450488618880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573207.750306:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488618880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573207.750318:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573207.750321:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573207.750323:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488618880:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573208.261297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573208.709375:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573209.285272:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573209.733411:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573210.309275:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573210.757324:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573211.333349:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573211.781438:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573212.357267:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573212.805330:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573212.869331:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573212.869343:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573212.869348:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573212.869360:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4361 00000100:00080000:3.0:1788573212.869371:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573212.869380:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573212.869423:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488619008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573212.869458:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573212.869463:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573212.869465:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573212.869477:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4361 00000100:00080000:3.0:1788573212.869482:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573212.869486:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573212.869495:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573212.869510:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488619136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573212.869525:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488619008:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573212.869525:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573212.869535:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573212.869538:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573212.869543:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4361 00000100:00080000:3.0:1788573212.869547:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573212.869552:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573212.869558:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573212.869574:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488619264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573212.869587:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788573212.869588:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573212.869594:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573212.869597:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573212.869604:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4361 00000100:00080000:3.0:1788573212.869609:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573212.869614:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573212.869620:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573212.869633:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573212/real 1788573212] req@ffff94d408c56680 x1875450488619008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573212.869635:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488619392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573212.869650:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573212.869655:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573212.869660:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573212.869669:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4361 00000100:00080000:3.0:1788573212.869674:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573212.869678:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488619008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573212.869682:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573212.869692:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788573212.869695:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573212.869700:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573212.869704:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488619008:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573212.869709:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488619520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573212.869728:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488619136:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573212.869756:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573212.869770:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573212/real 1788573212] req@ffff94d408c54a80 x1875450488619136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573212.869784:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488619136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573212.869790:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573212.869792:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573212.869796:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488619136:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573212.869803:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488619264:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573212.869826:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573212.869839:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573212/real 1788573212] req@ffff94d408c56d80 x1875450488619264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573212.869849:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488619264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573212.869859:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573212.869862:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573212.869864:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488619264:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573212.869868:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488619392:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573212.869883:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573212.869889:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573212/real 1788573212] req@ffff94d408c57b80 x1875450488619392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573212.869897:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488619392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573212.869916:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573212.869919:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573212.869922:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488619392:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573212.869926:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488619520:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573212.869940:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573212.869947:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573212/real 1788573212] req@ffff94d408c54e00 x1875450488619520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573212.869955:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488619520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573212.869960:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573212.869962:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573212.869964:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488619520:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573213.381327:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573213.829418:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573214.405278:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573214.853459:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573215.429352:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573215.877381:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573216.453299:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573216.901375:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573217.477313:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573217.925383:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573217.989315:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573217.989331:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573217.989337:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573217.989350:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4366 00000100:00080000:0.0:1788573217.989363:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573217.989371:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573217.989426:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488619648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573217.989469:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573217.989475:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573217.989477:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573217.989481:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4366 00000100:00080000:0.0:1788573217.989486:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573217.989491:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573217.989501:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573217.989515:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488619648:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573217.989517:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f800 x1875450488619776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573217.989541:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573217.989544:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573217.989545:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573217.989551:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4366 00000100:00080000:0.0:1788573217.989557:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573217.989562:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573217.989568:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573217.989588:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488619904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573217.989600:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573217.989604:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573217.989607:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573217.989615:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4366 00000100:00080000:0.0:1788573217.989622:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573217.989627:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573217.989634:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573217.989640:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488619776:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573217.989651:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f100 x1875450488620032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573217.989666:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573217.989670:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573217.989672:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573217.989677:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4366 00000100:00100000:3.0:1788573217.989677:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488619904:192.168.202.125@tcp:38:kworker.0 00000100:00080000:0.0:1788573217.989690:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573217.989697:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573217.989705:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573217.989718:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488620032:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573217.989724:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488620160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573217.989753:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488620160:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573218.501306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573218.949348:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573219.525322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573219.973365:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573220.549281:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573220.997324:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788573221.061462:0:2392:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:1.0:1788573221.061520:0:2392:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788573221.061537:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788573221.061587:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788573221.061599:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788573221.061612:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788573221.061623:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:3.0:1788573221.061637:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573217/real 1788573221] req@ffff94d408d3f100 x1875450488620032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573221.061667:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f100 x1875450488620032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573221.061679:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573221.061685:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573221.061689:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488620032:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573221.061697:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573217/real 1788573221] req@ffff94d408d3f800 x1875450488619776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573221.061710:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f800 x1875450488619776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573221.061718:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573221.061721:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573221.061724:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488619776:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573221.061731:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573217/real 1788573221] req@ffff94d408d3ca80 x1875450488619904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573221.061742:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488619904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573221.061764:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573221.061768:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573221.061773:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488619904:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573221.061780:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573217/real 1788573221] req@ffff94d408d3d500 x1875450488619648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573221.061793:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488619648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573221.061800:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573221.061804:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573221.061808:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488619648:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573221.061838:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573217/real 1788573221] req@ffff94d408d3e680 x1875450488620160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573221.061851:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488620160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573221.061858:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573221.061861:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573221.061865:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488620160:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573221.573298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573222.021365:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573222.597314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573223.045358:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573223.109337:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573223.109345:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573223.109349:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573223.109357:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4371 00000100:00080000:3.0:1788573223.109367:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573223.109373:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573223.109409:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488620288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573223.109438:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573223.109441:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573223.109443:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573223.109446:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4371 00000100:00080000:3.0:1788573223.109448:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573223.109450:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573223.109457:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573223.109470:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488620416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573223.109482:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573223.109486:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488620288:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573223.109488:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573223.109493:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573223.109500:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4371 00000100:00080000:3.0:1788573223.109504:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573223.109509:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573223.109515:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573223.109529:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488620544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573223.109539:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573223.109543:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573223.109546:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573223.109550:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4371 00000100:00080000:3.0:1788573223.109555:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573223.109561:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573223.109566:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573223.109578:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573223.109580:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488620672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573223.109592:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573223.109595:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573223.109598:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573223.109603:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4371 00000100:00080000:3.0:1788573223.109607:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573223.109611:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573223.109617:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573223.109628:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573223/real 1788573223] req@ffff94d408c54380 x1875450488620288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573223.109631:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488620800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573223.109672:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488620288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573223.109684:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573223.109689:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573223.109693:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488620288:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573223.109740:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488620416:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573223.109768:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573223.109784:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573223/real 1788573223] req@ffff94d408c57100 x1875450488620416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573223.109798:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488620416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573223.109808:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573223.109811:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573223.109814:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488620416:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573223.109820:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488620544:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573223.109845:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573223.109857:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573223/real 1788573223] req@ffff94d408c57800 x1875450488620544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573223.109869:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488620544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573223.109877:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573223.109880:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573223.109883:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488620544:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573223.109889:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488620672:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573223.109911:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573223.109924:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573223/real 1788573223] req@ffff94d408c55c00 x1875450488620672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573223.109936:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488620672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573223.109943:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573223.109946:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573223.109949:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488620672:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573223.109954:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488620800:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573223.109972:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573223.109983:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573223/real 1788573223] req@ffff94d408c55880 x1875450488620800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573223.109994:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488620800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573223.110007:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573223.110010:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573223.110013:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488620800:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573223.621359:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573224.069407:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573224.645310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573225.093394:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573225.669289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573226.117361:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573226.693361:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573227.141350:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573227.717354:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573228.165403:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573228.229318:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573228.229326:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573228.229330:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573228.229338:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4376 00000100:00080000:3.0:1788573228.229349:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573228.229357:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573228.229396:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488620928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573228.229432:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573228.229435:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573228.229437:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573228.229440:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4376 00000100:00080000:3.0:1788573228.229443:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573228.229446:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573228.229456:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573228.229470:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488621056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573228.229481:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573228.229486:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573228.229488:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573228.229495:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4376 00000100:00100000:0.0:1788573228.229498:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488620928:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573228.229500:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573228.229509:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573228.229514:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573228.229525:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488621184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573228.229534:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573228.229537:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573228.229538:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573228.229541:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4376 00000100:00080000:3.0:1788573228.229545:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573228.229547:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573228.229551:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573228.229559:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488621312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573228.229567:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573228.229572:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573228.229577:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573228.229579:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573228.229585:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4376 00000100:00080000:3.0:1788573228.229591:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573228.229595:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573228.229602:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573228.229608:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573228/real 1788573228] req@ffff94d408c56a00 x1875450488620928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573228.229640:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488621440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573228.229675:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488620928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573228.229685:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573228.229689:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573228.229692:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488620928:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573228.229699:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488621056:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573228.229718:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573228.229728:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573228/real 1788573228] req@ffff94d408c55180 x1875450488621056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573228.229736:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488621056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573228.229742:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573228.229744:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573228.229746:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488621056:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573228.229764:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488621184:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573228.229775:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573228.229781:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573228/real 1788573228] req@ffff94d408c54000 x1875450488621184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573228.229788:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488621184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573228.229794:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573228.229796:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573228.229798:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488621184:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573228.229801:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488621312:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573228.229811:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573228.229816:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573228/real 1788573228] req@ffff94d408c56300 x1875450488621312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573228.229823:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488621312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573228.229828:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573228.229830:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573228.229832:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488621312:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573228.229835:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488621440:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573228.229845:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573228.229850:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573228/real 1788573228] req@ffff94d408c57480 x1875450488621440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573228.229857:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488621440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573228.229862:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573228.229864:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573228.229866:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488621440:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573228.741424:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573229.189388:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573229.765513:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573230.213329:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573230.789372:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573231.237427:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573231.814288:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573232.261360:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573232.837303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573233.285423:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573233.349302:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573233.349311:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573233.349315:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573233.349325:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4381 00000100:00080000:3.0:1788573233.349336:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573233.349343:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573233.349380:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488621568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573233.349415:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573233.349419:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573233.349421:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573233.349425:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4381 00000100:00080000:3.0:1788573233.349429:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573233.349433:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573233.349442:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573233.349455:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488621696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573233.349465:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573233.349469:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573233.349471:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573233.349475:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4381 00000100:00080000:3.0:1788573233.349480:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573233.349485:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573233.349491:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573233.349499:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488621568:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573233.349512:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488621824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573233.349543:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573233.349547:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573233.349550:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573233.349555:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4381 00000100:00080000:3.0:1788573233.349560:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573233.349565:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573233.349574:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573233.349591:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488621952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573233.349601:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788573233.349603:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573233.349610:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573233.349613:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573233.349619:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4381 00000100:00080000:3.0:1788573233.349624:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573233.349629:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573233.349635:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573233.349651:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488622080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573233.349656:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573233/real 1788573233] req@ffff94d408c54700 x1875450488621568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573233.349710:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488621568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573233.349721:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573233.349725:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573233.349730:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488621568:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573233.349739:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488621696:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573233.349757:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573233.349765:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573233/real 1788573233] req@ffff94d408c55f80 x1875450488621696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573233.349778:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488621696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573233.349784:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573233.349786:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573233.349788:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488621696:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573233.349813:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488621824:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573233.349825:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573233.349831:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573233/real 1788573233] req@ffff94d408c57480 x1875450488621824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573233.349839:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488621824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573233.349845:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573233.349846:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573233.349848:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488621824:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573233.349852:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488621952:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573233.349863:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573233.349868:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573233/real 1788573233] req@ffff94d408c56300 x1875450488621952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573233.349876:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488621952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573233.349885:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573233.349887:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573233.349889:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488621952:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573233.349893:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488622080:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573233.349903:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573233.349908:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573233/real 1788573233] req@ffff94d408c54000 x1875450488622080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573233.349914:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488622080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573233.349919:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573233.349921:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573233.349923:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488622080:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573233.861296:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573234.309374:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573234.885331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573235.333380:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573235.909284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573236.357392:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573236.933278:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573237.381380:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573237.957267:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573238.405470:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573238.469282:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573238.469297:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573238.469301:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573238.469313:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4386 00000100:00080000:3.0:1788573238.469324:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573238.469331:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573238.469365:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488622208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573238.469395:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573238.469400:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573238.469401:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573238.469407:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4386 00000100:00080000:3.0:1788573238.469412:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573238.469438:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488622208:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573238.469442:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573238.469455:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573238.469472:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488622336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573238.469485:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573238.469490:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573238.469491:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573238.469505:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4386 00000100:00080000:3.0:1788573238.469510:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788573238.469515:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573238.469520:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573238.469527:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573238.469544:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488622464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573238.469555:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573238.469557:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573238/real 1788573238] req@ffff94d408c55180 x1875450488622208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573238.469561:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573238.469564:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573238.469572:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4386 00000100:00080000:3.0:1788573238.469578:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573238.469591:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573238.469593:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488622208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788573238.469597:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573238.469609:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488622592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573238.469612:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573238.469622:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573238.469624:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:3.0:1788573238.469625:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573238.469628:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573238.469634:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4386 00000100:00100000:1.0:1788573238.469635:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488622208:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573238.469639:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573238.469644:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573238.469647:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573238.469654:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488622720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573238.469668:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488622336:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573238.469711:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573238.469727:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573238/real 1788573238] req@ffff94d408c56a00 x1875450488622336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573238.469741:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488622336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573238.469750:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573238.469763:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573238.469770:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488622336:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573238.469777:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488622464:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573238.469799:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573238.469813:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573238/real 1788573238] req@ffff94d408c55880 x1875450488622464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573238.469836:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488622464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573238.469844:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573238.469848:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573238.469851:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488622464:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573238.469858:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488622592:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573238.469880:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573238.469890:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573238/real 1788573238] req@ffff94d408c55c00 x1875450488622592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573238.469904:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488622592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573238.469911:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573238.469914:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573238.469918:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488622592:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573238.469924:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488622720:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573238.469946:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573238.469956:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573238/real 1788573238] req@ffff94d408c57800 x1875450488622720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573238.469964:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488622720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573238.469974:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573238.469976:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573238.469978:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488622720:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573238.981307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573239.429482:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573240.005257:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573240.453443:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573241.029265:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573241.477354:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573242.053284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573242.501368:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573243.077302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573243.525392:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573243.589267:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573243.589275:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573243.589279:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573243.589288:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4391 00000100:00080000:3.0:1788573243.589297:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573243.589304:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573243.589349:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488622848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573243.589374:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573243.589377:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573243.589380:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573243.589383:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4391 00000100:00080000:3.0:1788573243.589386:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573243.589390:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573243.589398:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573243.589411:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488622976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573243.589419:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573243.589421:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573243.589424:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573243.589429:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488622848:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573243.589430:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4391 00000100:00080000:3.0:1788573243.589434:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573243.589439:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573243.589443:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573243.589455:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488623104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573243.589462:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573243.589464:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573243.589466:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573243.589469:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4391 00000100:00080000:3.0:1788573243.589472:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573243.589475:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573243.589480:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573243.589493:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488623232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573243.589504:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573243.589508:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573243.589510:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573243.589515:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4391 00000800:00000100:1.0:1788573243.589521:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573243.589521:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573243.589529:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573243.589537:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573243.589550:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488623360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573243.589563:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573243/real 1788573243] req@ffff94d408c57100 x1875450488622848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573243.589579:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488622848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573243.589587:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573243.589591:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573243.589602:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488622848:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573243.589632:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488622976:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573243.589659:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573243.589670:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573243/real 1788573243] req@ffff94d408c54380 x1875450488622976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573243.589679:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488622976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573243.589685:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573243.589687:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573243.589690:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488622976:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573243.589695:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488623104:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573243.589707:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573243.589714:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573243/real 1788573243] req@ffff94d408c54e00 x1875450488623104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573243.589722:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488623104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573243.589727:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573243.589729:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573243.589731:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488623104:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573243.589735:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488623232:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573243.589746:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573243.589752:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573243/real 1788573243] req@ffff94d408c57b80 x1875450488623232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573243.589759:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488623232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573243.589764:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573243.589766:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573243.589768:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488623232:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573243.589771:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488623360:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573243.589781:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573243.589788:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573243/real 1788573243] req@ffff94d408c56d80 x1875450488623360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573243.589795:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488623360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573243.589800:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573243.589802:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573243.589805:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488623360:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573244.101313:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573244.549353:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573245.125325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573245.573422:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573246.149349:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573246.597342:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573247.173347:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573247.621364:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573248.197332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573248.645384:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573248.709304:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573248.709323:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573248.709327:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573248.709337:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4397 00000100:00080000:3.0:1788573248.709348:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573248.709355:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573248.709388:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488623488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573248.709411:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573248.709415:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573248.709416:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573248.709420:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4397 00000100:00080000:3.0:1788573248.709422:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573248.709425:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573248.709433:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573248.709441:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488623616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573248.709449:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573248.709451:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573248.709453:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573248.709456:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488623488:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573248.709458:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4397 00000100:00080000:3.0:1788573248.709462:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573248.709466:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573248.709469:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573248.709479:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488623744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573248.709485:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573248.709487:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573248.709488:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573248.709492:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4397 00000100:00080000:3.0:1788573248.709495:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573248.709497:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573248.709500:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573248.709507:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488623872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573248.709514:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788573248.709517:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573248.709519:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573248.709522:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573248.709525:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4397 00000100:00080000:3.0:1788573248.709528:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573248.709531:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573248.709534:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573248.709546:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488624000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573248.709551:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573248/real 1788573248] req@ffff94d408c54a80 x1875450488623488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573248.709572:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488623488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573248.709581:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573248.709584:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573248.709587:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488623488:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573248.709606:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488623616:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573248.709621:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573248.709629:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573248/real 1788573248] req@ffff94d408c56680 x1875450488623616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573248.709650:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488623616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573248.709658:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573248.709661:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573248.709663:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488623616:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573248.709667:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488623744:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573248.709679:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573248.709686:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573248/real 1788573248] req@ffff94d408c56d80 x1875450488623744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573248.709694:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488623744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573248.709700:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573248.709702:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573248.709704:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488623744:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573248.709707:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488623872:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573248.709717:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573248.709723:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573248/real 1788573248] req@ffff94d408c57b80 x1875450488623872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573248.709730:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488623872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573248.709740:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573248.709742:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573248.709744:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488623872:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573248.709747:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488624000:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573248.709756:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573248.709761:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573248/real 1788573248] req@ffff94d408c54e00 x1875450488624000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573248.709768:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488624000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573248.709773:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573248.709774:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573248.709776:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488624000:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573249.221290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573249.669397:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573250.245372:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573250.693410:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573251.269304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573251.717341:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573252.293381:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573252.741450:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573253.317294:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573253.765385:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573253.829273:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573253.829282:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573253.829285:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573253.829294:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4402 00000100:00080000:3.0:1788573253.829307:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573253.829314:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573253.829347:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488624128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573253.829370:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573253.829373:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573253.829374:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573253.829377:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4402 00000100:00080000:3.0:1788573253.829379:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573253.829383:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573253.829389:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573253.829399:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488624256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573253.829406:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573253.829409:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573253.829410:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573253.829415:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488624128:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573253.829415:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4402 00000100:00080000:3.0:1788573253.829419:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573253.829425:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573253.829428:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573253.829437:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488624384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573253.829443:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573253.829445:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573253.829446:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573253.829450:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4402 00000100:00080000:3.0:1788573253.829452:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573253.829455:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573253.829458:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573253.829465:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488624512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573253.829471:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573253.829473:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573253.829475:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788573253.829479:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573253.829482:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4402 00000100:00080000:3.0:1788573253.829486:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573253.829489:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573253.829492:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573253.829500:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488624640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573253.829519:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573253/real 1788573253] req@ffff94d408c54380 x1875450488624128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573253.829533:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488624128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573253.829541:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573253.829544:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573253.829547:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488624128:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573253.829565:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488624256:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573253.829581:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573253.829589:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573253/real 1788573253] req@ffff94d408c57100 x1875450488624256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573253.829597:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488624256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573253.829602:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573253.829604:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573253.829607:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488624256:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573253.829610:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488624384:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573253.829620:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573253.829628:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573253/real 1788573253] req@ffff94d408c57800 x1875450488624384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573253.829640:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488624384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573253.829649:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573253.829653:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573253.829656:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488624384:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573253.829663:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488624512:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573253.829692:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573253.829700:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573253/real 1788573253] req@ffff94d408c55c00 x1875450488624512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573253.829708:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488624512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573253.829714:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573253.829716:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573253.829718:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488624512:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573253.829721:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488624640:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573253.829732:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573253.829740:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573253/real 1788573253] req@ffff94d408c55880 x1875450488624640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573253.829752:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488624640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573253.829761:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573253.829764:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573253.829768:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488624640:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573254.341339:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573254.789481:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573255.365320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573255.813344:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573256.389332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573256.837408:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573257.413300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573257.861322:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573258.437315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573258.885416:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573258.949325:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573258.949334:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573258.949338:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573258.949353:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4407 00000100:00080000:0.0:1788573258.949363:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573258.949370:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573258.949402:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488624768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573258.949423:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573258.949426:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573258.949428:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573258.949430:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4407 00000100:00080000:0.0:1788573258.949432:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573258.949435:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573258.949442:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573258.949451:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488624896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573258.949458:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573258.949461:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573258.949462:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573258.949465:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4407 00000100:00080000:0.0:1788573258.949467:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573258.949471:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488624768:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788573258.949472:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573258.949479:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573258.949487:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488625024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573258.949499:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573258.949502:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573258.949503:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573258.949506:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4407 00000100:00080000:0.0:1788573258.949508:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573258.949510:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573258.949514:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573258.949520:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488625152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573258.949527:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573258.949530:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573258.949531:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573258.949533:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4407 00000100:00080000:0.0:1788573258.949535:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573258.949537:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573258.949540:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573258.949543:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573258.949550:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488625280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573258.949586:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573258/real 1788573258] req@ffff94d408d3ea00 x1875450488624768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573258.949612:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488624768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573258.949622:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573258.949625:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573258.949629:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488624768:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573258.949636:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488624896:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573258.949657:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573258.949669:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573258/real 1788573258] req@ffff94d408d3ce00 x1875450488624896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573258.949682:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488624896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573258.949690:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573258.949693:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573258.949696:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488624896:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573258.949720:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488625024:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573258.949738:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573258.949747:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573258/real 1788573258] req@ffff94d408d3df80 x1875450488625024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573258.949755:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488625024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573258.949765:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573258.949767:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573258.949769:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488625024:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573258.949773:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488625152:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573258.949784:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573258.949790:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573258/real 1788573258] req@ffff94d408d3d180 x1875450488625152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573258.949797:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488625152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573258.949802:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573258.949804:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573258.949805:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488625152:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573258.949809:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488625280:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573258.949818:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573258.949823:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573258/real 1788573258] req@ffff94d408d3c000 x1875450488625280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573258.949830:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488625280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573258.949835:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573258.949837:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573258.949839:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488625280:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573259.461296:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573259.909334:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573260.485311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573260.933389:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573261.509325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573261.957332:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573262.533302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573262.981394:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573263.557300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573264.005312:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573264.069294:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573264.069305:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573264.069310:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573264.069320:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4412 00000100:00080000:3.0:1788573264.069332:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573264.069340:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573264.069378:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488625408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573264.069402:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573264.069405:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573264.069406:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573264.069409:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4412 00000100:00080000:3.0:1788573264.069412:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573264.069416:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573264.069424:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573264.069433:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488625536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573264.069445:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573264.069450:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788573264.069453:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488625408:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573264.069453:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573264.069464:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4412 00000100:00080000:3.0:1788573264.069469:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573264.069473:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573264.069479:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573264.069495:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488625664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573264.069502:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573264.069505:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573264.069507:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573264.069510:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4412 00000100:00080000:3.0:1788573264.069513:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573264.069516:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:1.0:1788573264.069521:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:3.0:1788573264.069523:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573264.069534:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488625792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573264.069541:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573264.069543:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573264.069545:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573264.069547:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4412 00000100:00080000:3.0:1788573264.069549:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573264.069552:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573264.069556:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573264.069563:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488625920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573264.069569:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573264/real 1788573264] req@ffff94d408c56a00 x1875450488625408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573264.069591:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488625408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573264.069603:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573264.069607:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573264.069612:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488625408:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573264.069639:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488625536:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573264.069661:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573264.069672:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573264/real 1788573264] req@ffff94d408c55180 x1875450488625536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573264.069685:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488625536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573264.069695:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573264.069698:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573264.069702:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488625536:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573264.069708:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488625664:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573264.069724:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573264.069733:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573264/real 1788573264] req@ffff94d408c54000 x1875450488625664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573264.069745:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488625664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573264.069755:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573264.069758:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573264.069762:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488625664:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573264.069767:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488625792:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573264.069782:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573264.069791:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573264/real 1788573264] req@ffff94d408c56300 x1875450488625792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573264.069804:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488625792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573264.069812:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573264.069815:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573264.069819:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488625792:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573264.069824:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488625920:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573264.069838:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573264.069846:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573264/real 1788573264] req@ffff94d408c57480 x1875450488625920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573264.069865:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488625920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573264.069874:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573264.069877:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573264.069880:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488625920:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573264.581307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573265.029343:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573265.605331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573266.053332:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573266.629279:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573267.077344:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573267.653284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573268.101371:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573268.677338:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573269.125351:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573269.189348:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573269.189361:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573269.189365:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573269.189376:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4417 00000100:00080000:3.0:1788573269.189390:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573269.189398:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573269.189446:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488626048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573269.189483:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573269.189489:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573269.189494:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573269.189498:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4417 00000100:00080000:3.0:1788573269.189501:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573269.189505:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573269.189515:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573269.189528:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488626176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573269.189538:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573269.189541:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573269.189543:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573269.189548:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4417 00000100:00100000:1.0:1788573269.189550:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488626048:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573269.189551:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573269.189558:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573269.189562:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573269.189573:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488626304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573269.189580:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573269.189582:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573269.189584:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573269.189586:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4417 00000100:00080000:3.0:1788573269.189588:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573269.189591:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573269.189594:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573269.189601:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488626432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573269.189607:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573269.189610:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573269.189611:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573269.189614:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4417 00000100:00080000:3.0:1788573269.189624:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573269.189627:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573269.189631:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573269.189634:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573269.189643:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488626560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573269.189673:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573269/real 1788573269] req@ffff94d408c55f80 x1875450488626048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573269.189691:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488626048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573269.189701:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573269.189704:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573269.189707:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488626048:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573269.189731:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488626176:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573269.189748:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573269.189757:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573269/real 1788573269] req@ffff94d408c54700 x1875450488626176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573269.189765:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488626176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573269.189772:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573269.189774:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573269.189777:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488626176:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573269.189780:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488626304:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573269.189792:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573269.189798:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573269/real 1788573269] req@ffff94d408c57480 x1875450488626304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573269.189805:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488626304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573269.189819:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573269.189821:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573269.189823:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488626304:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573269.189827:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488626432:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573269.189841:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573269.189847:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573269/real 1788573269] req@ffff94d408c56300 x1875450488626432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573269.189855:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488626432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573269.189861:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573269.189863:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573269.189866:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488626432:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573269.189870:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488626560:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573269.189897:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573269.189905:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573269/real 1788573269] req@ffff94d408c54000 x1875450488626560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573269.189913:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488626560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573269.189919:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573269.189921:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573269.189923:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488626560:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573269.701338:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573270.149436:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573270.725299:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573271.173363:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573271.749364:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573272.197408:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573272.773263:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573273.221351:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573273.797335:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573274.245369:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573274.309392:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573274.309409:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573274.309414:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573274.309425:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4422 00000100:00080000:3.0:1788573274.309435:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573274.309442:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573274.309476:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488626688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573274.309504:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573274.309508:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573274.309510:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573274.309514:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4422 00000100:00080000:3.0:1788573274.309518:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573274.309523:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573274.309531:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573274.309538:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488626688:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573274.309547:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488626816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573274.309559:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573274.309563:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573274.309565:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573274.309577:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4422 00000100:00080000:3.0:1788573274.309582:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788573274.309586:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573274.309587:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573274.309594:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573274.309609:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488626944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573274.309619:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573274.309622:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573274/real 1788573274] req@ffff94d408c55180 x1875450488626688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573274.309623:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573274.309627:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573274.309636:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4422 00000100:00080000:3.0:1788573274.309641:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573274.309645:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488626688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573274.309649:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00080000:1.0:1788573274.309659:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788573274.309659:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788573274.309664:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573274.309668:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488626688:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573274.309676:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488627072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573274.309687:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488626816:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788573274.309688:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573274.309697:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573274.309699:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573274.309703:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4422 00000100:00080000:3.0:1788573274.309708:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788573274.309718:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573274.309727:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573274.309728:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573274/real 1788573274] req@ffff94d408c56a00 x1875450488626816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788573274.309736:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573274.309740:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488626816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573274.309747:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788573274.309753:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488627200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573274.309755:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573274.309760:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488626816:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573274.309768:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488626944:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573274.309780:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573274.309789:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573274/real 1788573274] req@ffff94d408c55880 x1875450488626944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573274.309796:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488626944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573274.309802:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573274.309804:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573274.309806:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488626944:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573274.309824:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488627072:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573274.309837:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573274.309843:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573274/real 1788573274] req@ffff94d408c55c00 x1875450488627072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573274.309868:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488627072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573274.309876:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573274.309879:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573274.309882:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488627072:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573274.309887:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488627200:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573274.309904:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573274.309914:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573274/real 1788573274] req@ffff94d408c57800 x1875450488627200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573274.309925:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488627200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573274.309932:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573274.309935:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573274.309938:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488627200:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573274.821321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573275.269381:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573275.845307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573276.293374:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573276.869329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573277.317363:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573277.893272:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573278.341377:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573278.917316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573279.365378:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573279.429288:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573279.429299:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573279.429303:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573279.429310:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4427 00000100:00080000:3.0:1788573279.429319:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573279.429325:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573279.429354:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488627328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573279.429373:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573279.429376:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573279.429377:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573279.429380:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4427 00000100:00080000:3.0:1788573279.429382:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573279.429384:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573279.429391:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573279.429401:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488627456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573279.429413:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573279.429416:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488627328:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573279.429416:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573279.429420:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573279.429427:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4427 00000100:00080000:3.0:1788573279.429430:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573279.429432:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573279.429436:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573279.429445:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488627584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573279.429452:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573279.429454:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573279.429456:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573279.429459:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4427 00000100:00080000:3.0:1788573279.429462:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573279.429465:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573279.429468:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573279.429474:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488627712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573279.429480:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573279.429482:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573279.429483:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573279.429485:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4427 00000100:00080000:3.0:1788573279.429488:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573279.429490:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:1.0:1788573279.429495:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:3.0:1788573279.429495:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573279.429530:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573279/real 1788573279] req@ffff94d408c57100 x1875450488627328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573279.429547:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488627328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573279.429557:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788573279.429561:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488627840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573279.429562:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573279.429570:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488627328:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573279.429606:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488627456:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573279.429631:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573279.429642:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573279/real 1788573279] req@ffff94d408c54380 x1875450488627456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573279.429650:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488627456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573279.429655:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573279.429657:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573279.429659:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488627456:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573279.429663:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488627584:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573279.429677:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573279.429683:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573279/real 1788573279] req@ffff94d408c54e00 x1875450488627584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573279.429696:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488627584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573279.429704:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573279.429708:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573279.429711:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488627584:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573279.429719:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488627712:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573279.429742:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573279.429766:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573279/real 1788573279] req@ffff94d408c57b80 x1875450488627712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573279.429777:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488627712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573279.429785:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573279.429788:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573279.429790:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488627712:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573279.429795:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488627840:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573279.429811:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573279.429818:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573279/real 1788573279] req@ffff94d408c56d80 x1875450488627840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573279.429824:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488627840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573279.429828:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573279.429830:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573279.429832:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488627840:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573279.941323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573280.389384:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573280.965326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573281.413348:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573281.989303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573282.437383:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573283.013269:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573283.461383:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573284.037305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573284.485360:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573284.549294:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573284.549303:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573284.549307:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573284.549316:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4432 00000100:00080000:3.0:1788573284.549326:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573284.549333:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573284.549365:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488627968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573284.549391:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573284.549394:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573284.549396:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573284.549399:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4432 00000100:00080000:3.0:1788573284.549402:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573284.549406:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573284.549414:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573284.549429:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488627968:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573284.549429:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488628096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573284.549448:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573284.549452:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573284.549454:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573284.549458:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4432 00000100:00080000:3.0:1788573284.549461:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573284.549464:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573284.549469:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573284.549480:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488628224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573284.549490:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573284.549493:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573284.549495:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573284.549499:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4432 00000100:00080000:3.0:1788573284.549503:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573284.549507:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573284.549513:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573284.549517:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488628096:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573284.549525:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488628352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573284.549538:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573284.549542:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1788573284.549545:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488628224:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788573284.549546:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573284.549554:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4432 00000100:00080000:3.0:1788573284.549558:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573284.549563:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573284.549568:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488628352:192.168.202.125@tcp:8:kworker.0 00000008:00100000:3.0:1788573284.549569:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573284.549583:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488628480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573284.549619:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488628480:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573285.061282:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573285.509318:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573286.085279:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573286.533382:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573287.109321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573287.557387:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788573287.621591:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:2.0:1788573287.621650:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788573287.621689:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788573287.621744:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788573287.621754:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788573287.621764:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788573287.621773:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:1.0:1788573287.621821:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573284/real 1788573287] req@ffff94d408c54e00 x1875450488628480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573287.621857:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488628480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573287.621872:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573287.621877:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573287.621883:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488628480:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573287.621892:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573284/real 1788573287] req@ffff94d408c57b80 x1875450488628352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573287.621906:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488628352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573287.621926:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573287.621930:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573287.621933:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488628352:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573287.621939:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573284/real 1788573287] req@ffff94d408c54a80 x1875450488628096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573287.621953:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488628096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573287.621961:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573287.621965:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573287.621969:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488628096:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573287.621975:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573284/real 1788573287] req@ffff94d408c56d80 x1875450488628224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573287.621987:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488628224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573287.621994:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573287.621998:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573287.622001:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488628224:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573287.622013:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573284/real 1788573287] req@ffff94d408c56680 x1875450488627968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573287.622027:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488627968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573287.622035:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573287.622039:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573287.622041:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488627968:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788573288.133299:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573288.581358:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573289.157265:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573289.605341:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573289.669281:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573289.669290:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573289.669292:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573289.669300:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4437 00000100:00080000:3.0:1788573289.669309:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573289.669315:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573289.669341:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488628608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573289.669366:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573289.669369:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573289.669371:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573289.669375:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4437 00000100:00080000:3.0:1788573289.669378:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573289.669381:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573289.669390:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573289.669401:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488628736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573289.669410:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573289.669413:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573289.669415:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573289.669418:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4437 00000100:00080000:3.0:1788573289.669421:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573289.669424:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573289.669429:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573289.669440:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488628864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573289.669448:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573289.669451:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573289.669453:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573289.669455:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4437 00000100:00080000:3.0:1788573289.669457:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573289.669460:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573289.669464:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573289.669474:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488628992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573289.669482:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573289.669485:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573289.669487:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573289.669490:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4437 00000100:00080000:3.0:1788573289.669494:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573289.669498:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573289.669502:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573289.669513:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488629120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573289.669524:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488628608:192.168.202.125@tcp:250:kworker.0 00000800:00000100:1.0:1788573289.669605:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573289.669657:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573289/real 1788573289] req@ffff94d408c54380 x1875450488628608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573289.669675:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488628608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573289.669687:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573289.669691:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573289.669695:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488628608:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573289.669703:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488628736:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573289.669786:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573289.669801:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573289/real 1788573289] req@ffff94d408c57100 x1875450488628736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573289.669813:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488628736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573289.669821:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573289.669824:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573289.669827:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488628736:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573289.669840:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488628864:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573289.669859:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573289.669870:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573289/real 1788573289] req@ffff94d408c57800 x1875450488628864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573289.669888:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488628864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573289.669896:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573289.669904:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573289.669907:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488628864:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573289.669913:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488628992:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573289.669934:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573289.669945:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573289/real 1788573289] req@ffff94d408c55c00 x1875450488628992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573289.669956:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488628992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573289.669962:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573289.669965:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573289.669968:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488628992:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573289.669999:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488629120:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573289.670018:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573289.670028:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573289/real 1788573289] req@ffff94d408c55880 x1875450488629120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573289.670039:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488629120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573289.670049:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573289.670052:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573289.670054:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488629120:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573290.181269:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573290.629376:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573291.205316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573291.653410:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573292.229265:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573292.677314:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573293.253404:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573293.701351:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573294.277679:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573294.725402:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573294.789331:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573294.789342:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573294.789346:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573294.789359:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4443 00000100:00080000:3.0:1788573294.789371:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573294.789380:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573294.789432:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488629248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573294.789460:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573294.789468:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573294.789470:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573294.789473:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4443 00000100:00080000:3.0:1788573294.789476:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573294.789479:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573294.789488:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573294.789498:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488629376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573294.789506:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573294.789509:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573294.789511:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573294.789514:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4443 00000100:00080000:3.0:1788573294.789518:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573294.789521:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573294.789524:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573294.789533:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488629248:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573294.789535:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488629504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573294.789549:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573294.789552:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573294.789553:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573294.789555:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4443 00000100:00080000:3.0:1788573294.789557:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573294.789559:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573294.789563:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573294.789582:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488629632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573294.789588:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573294.789590:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573294.789591:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573294.789594:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4443 00000100:00080000:3.0:1788573294.789596:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573294.789598:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573294.789602:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573294.789609:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488629760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573294.789642:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573294.789691:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573294/real 1788573294] req@ffff94d408c56a00 x1875450488629248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573294.789713:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488629248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573294.789726:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573294.789731:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573294.789736:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488629248:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573294.789745:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488629376:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573294.789770:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573294.789785:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573294/real 1788573294] req@ffff94d408c55180 x1875450488629376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573294.789799:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488629376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573294.789807:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573294.789812:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573294.789815:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488629376:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573294.789846:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488629504:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573294.789868:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573294.789880:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573294/real 1788573294] req@ffff94d408c54000 x1875450488629504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573294.789894:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488629504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573294.789914:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573294.789919:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573294.789927:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488629504:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573294.789934:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488629632:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573294.789957:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573294.789969:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573294/real 1788573294] req@ffff94d408c56300 x1875450488629632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573294.789982:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488629632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573294.789991:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573294.789995:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573294.789999:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488629632:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573294.790006:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488629760:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573294.790026:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573294.790038:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573294/real 1788573294] req@ffff94d408c57480 x1875450488629760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573294.790051:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488629760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573294.790059:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573294.790063:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573294.790067:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488629760:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573295.301413:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573295.749450:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573296.325427:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573296.773440:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573297.350123:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573297.797464:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573298.373368:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573298.821370:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573299.397319:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573299.845364:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573299.909252:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573299.909268:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573299.909271:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573299.909280:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4448 00000100:00080000:3.0:1788573299.909289:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573299.909296:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573299.909328:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488629888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573299.909349:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573299.909353:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573299.909354:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573299.909357:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4448 00000100:00080000:3.0:1788573299.909360:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573299.909363:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573299.909370:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573299.909379:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488630016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573299.909387:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488629888:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573299.909392:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573299.909396:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573299.909397:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573299.909400:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4448 00000100:00080000:3.0:1788573299.909402:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573299.909406:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573299.909409:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573299.909419:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488630144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573299.909426:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573299.909429:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573299.909430:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573299.909432:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4448 00000100:00080000:3.0:1788573299.909435:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573299.909438:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573299.909448:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573299.909457:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488630272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573299.909462:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573299.909471:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573299.909476:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573299.909478:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573299.909483:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4448 00000100:00080000:3.0:1788573299.909488:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573299.909493:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573299.909498:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573299.909504:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573299/real 1788573299] req@ffff94d408c54700 x1875450488629888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573299.909534:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488630400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573299.909539:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488629888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573299.909562:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573299.909567:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573299.909571:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488629888:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573299.909602:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488630016:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573299.909624:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573299.909634:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573299/real 1788573299] req@ffff94d408c55f80 x1875450488630016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573299.909644:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488630016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573299.909650:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573299.909652:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573299.909654:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488630016:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573299.909666:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488630144:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573299.909685:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573299.909697:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573299/real 1788573299] req@ffff94d408c57480 x1875450488630144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573299.909711:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488630144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573299.909720:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573299.909724:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573299.909726:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488630144:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573299.909731:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488630272:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573299.909746:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573299.909754:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573299/real 1788573299] req@ffff94d408c56300 x1875450488630272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573299.909761:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488630272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573299.909771:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573299.909773:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573299.909775:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488630272:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573299.909778:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488630400:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573299.909788:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573299.909793:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573299/real 1788573299] req@ffff94d408c54000 x1875450488630400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573299.909800:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488630400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573299.909805:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573299.909807:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573299.909808:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488630400:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573300.421310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573300.869344:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573301.445300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573301.893389:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573302.469327:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573302.917336:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573303.493318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573303.941398:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573304.517343:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573304.965347:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573305.029257:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573305.029269:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573305.029274:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573305.029301:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4453 00000100:00080000:3.0:1788573305.029313:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573305.029320:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573305.029353:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488630528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573305.029376:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573305.029379:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573305.029381:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573305.029384:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4453 00000100:00080000:3.0:1788573305.029386:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573305.029389:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573305.029396:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573305.029405:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488630656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573305.029413:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573305.029416:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573305.029418:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573305.029422:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488630528:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573305.029424:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4453 00000100:00080000:3.0:1788573305.029431:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573305.029434:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573305.029438:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573305.029463:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488630784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573305.029471:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573305.029473:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573305.029474:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573305.029477:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4453 00000100:00080000:3.0:1788573305.029480:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573305.029483:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573305.029488:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573305.029498:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573305.029498:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488630912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573305.029508:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573305.029511:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573305.029513:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573305.029516:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4453 00000100:00080000:3.0:1788573305.029518:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573305.029520:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573305.029523:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573305.029536:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573305/real 1788573305] req@ffff94d408c55180 x1875450488630528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573305.029536:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488631040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573305.029578:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488630528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573305.029597:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573305.029602:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573305.029608:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488630528:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573305.029631:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488630656:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573305.029677:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573305.029691:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573305/real 1788573305] req@ffff94d408c56a00 x1875450488630656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573305.029706:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488630656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573305.029714:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573305.029718:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573305.029721:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488630656:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573305.029728:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488630784:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573305.029751:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573305.029763:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573305/real 1788573305] req@ffff94d408c55880 x1875450488630784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573305.029772:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488630784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573305.029778:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573305.029781:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573305.029785:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488630784:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573305.029792:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488630912:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573305.029813:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573305.029826:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573305/real 1788573305] req@ffff94d408c55c00 x1875450488630912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573305.029839:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488630912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573305.029845:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573305.029847:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573305.029849:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488630912:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573305.029853:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488631040:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573305.029871:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573305.029883:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573305/real 1788573305] req@ffff94d408c57800 x1875450488631040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573305.029895:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488631040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573305.029903:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573305.029906:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573305.029910:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488631040:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573305.541303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573305.989360:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573306.565303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573307.013374:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573307.589279:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573308.037371:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573308.613284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573309.061367:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573309.637285:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573310.085374:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573310.149317:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573310.149328:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573310.149332:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573310.149342:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4458 00000100:00080000:3.0:1788573310.149353:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573310.149361:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573310.149434:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488631168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573310.149463:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573310.149467:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573310.149469:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573310.149474:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4458 00000100:00080000:3.0:1788573310.149478:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573310.149483:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573310.149493:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573310.149506:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488631168:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573310.149509:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488631296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573310.149523:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573310.149527:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573310.149529:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573310.149533:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4458 00000100:00080000:3.0:1788573310.149538:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573310.149542:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573310.149547:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573310.149562:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488631424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573310.149574:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573310.149574:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573310.149580:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573310.149582:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573310.149588:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4458 00000100:00080000:3.0:1788573310.149595:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573310.149600:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573310.149606:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573310.149611:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573310/real 1788573310] req@ffff94d408c57100 x1875450488631168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573310.149622:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488631552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573310.149637:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488631168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573310.149638:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573310.149649:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573310.149654:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573310.149657:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573310.149661:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4458 00000100:00080000:1.0:1788573310.149664:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573310.149670:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488631168:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573310.149670:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573310.149678:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573310.149684:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573310.149695:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488631296:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573310.149698:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488631680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573310.149720:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573310.149731:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573310/real 1788573310] req@ffff94d408c54380 x1875450488631296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573310.149741:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488631296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573310.149747:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573310.149749:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573310.149751:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488631296:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573310.149755:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488631424:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573310.149765:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573310.149771:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573310/real 1788573310] req@ffff94d408c56680 x1875450488631424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573310.149779:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488631424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573310.149788:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573310.149790:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573310.149792:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488631424:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573310.149796:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488631552:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573310.149806:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573310.149811:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573310/real 1788573310] req@ffff94d408c56d80 x1875450488631552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573310.149818:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488631552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573310.149823:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573310.149825:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573310.149827:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488631552:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573310.149836:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488631680:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573310.149846:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573310.149860:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573310/real 1788573310] req@ffff94d408c54a80 x1875450488631680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573310.149869:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488631680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573310.149874:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573310.149876:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573310.149879:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488631680:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573310.661321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573311.109381:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573311.685299:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573312.133335:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573312.709339:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573313.157412:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573313.733268:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573314.181404:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573314.757320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573315.205408:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573315.269266:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573315.269279:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573315.269285:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573315.269295:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4463 00000100:00080000:3.0:1788573315.269307:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573315.269315:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573315.269375:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488631808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573315.269399:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573315.269402:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573315.269404:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573315.269407:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4463 00000100:00080000:3.0:1788573315.269410:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573315.269414:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573315.269421:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573315.269431:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488631936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573315.269440:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573315.269443:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573315.269444:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573315.269447:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4463 00000100:00080000:3.0:1788573315.269449:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573315.269451:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573315.269455:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573315.269458:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488631808:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573315.269466:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488632064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573315.269480:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573315.269483:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573315.269484:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573315.269487:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4463 00000100:00080000:3.0:1788573315.269491:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573315.269494:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573315.269498:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573315.269507:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488632192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573315.269514:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573315.269517:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1788573315.269521:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573315.269521:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573315.269527:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4463 00000100:00080000:3.0:1788573315.269531:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573315.269534:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573315.269538:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573315.269546:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488632320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573315.269557:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573315/real 1788573315] req@ffff94d408c57b80 x1875450488631808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573315.269587:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488631808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573315.269598:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573315.269602:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573315.269607:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488631808:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573315.269614:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488631936:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573315.269638:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573315.269655:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573315/real 1788573315] req@ffff94d408c54e00 x1875450488631936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573315.269671:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488631936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573315.269680:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573315.269684:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573315.269687:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488631936:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573315.269706:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488632064:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573315.269722:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573315.269730:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573315/real 1788573315] req@ffff94d408c54a80 x1875450488632064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573315.269738:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488632064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573315.269744:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573315.269746:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573315.269748:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488632064:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573315.269752:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488632192:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573315.269777:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573315.269783:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573315/real 1788573315] req@ffff94d408c56d80 x1875450488632192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573315.269791:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488632192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573315.269796:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573315.269799:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573315.269801:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488632192:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573315.269804:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488632320:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573315.269814:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573315.269821:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573315/real 1788573315] req@ffff94d408c56680 x1875450488632320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573315.269841:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488632320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573315.269850:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573315.269854:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573315.269857:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488632320:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573315.781355:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573316.229411:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573316.805338:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573317.253432:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573317.829375:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573318.277495:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573318.853355:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573319.301429:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573319.877353:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573320.325383:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573320.389325:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573320.389336:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573320.389341:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573320.389351:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4468 00000100:00080000:3.0:1788573320.389362:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573320.389369:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573320.389405:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488632448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573320.389433:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573320.389437:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573320.389438:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573320.389442:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4468 00000100:00080000:3.0:1788573320.389445:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573320.389449:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573320.389458:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573320.389471:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488632576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573320.389483:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573320.389484:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488632448:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573320.389489:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573320.389493:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573320.389498:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4468 00000100:00080000:3.0:1788573320.389501:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573320.389505:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573320.389511:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573320.389525:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488632704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573320.389540:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788573320.389545:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573320.389547:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573320.389551:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573320.389556:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4468 00000100:00080000:3.0:1788573320.389561:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573320.389566:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573320.389572:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573320.389579:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573320/real 1788573320] req@ffff94d408c54380 x1875450488632448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573320.389585:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488632832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573320.389598:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573320.389600:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488632448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573320.389602:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573320.389606:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573320.389613:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4468 00000100:00080000:3.0:1788573320.389616:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573320.389620:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00080000:1.0:1788573320.389625:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788573320.389626:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788573320.389631:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573320.389636:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488632448:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573320.389643:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488632960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573320.389659:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488632576:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573320.389679:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573320.389689:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573320/real 1788573320] req@ffff94d408c57100 x1875450488632576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573320.389697:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488632576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573320.389708:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573320.389710:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573320.389712:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488632576:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573320.389716:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488632704:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573320.389725:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573320.389731:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573320/real 1788573320] req@ffff94d408c57800 x1875450488632704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573320.389738:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488632704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573320.389744:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573320.389745:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573320.389747:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488632704:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573320.389750:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488632832:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573320.389760:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573320.389766:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573320/real 1788573320] req@ffff94d408c55c00 x1875450488632832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573320.389773:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488632832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573320.389778:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573320.389780:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573320.389782:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488632832:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573320.389786:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488632960:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573320.389795:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573320.389800:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573320/real 1788573320] req@ffff94d408c55880 x1875450488632960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573320.389806:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488632960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573320.389812:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573320.389813:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573320.389816:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488632960:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573320.901392:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573321.349479:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573321.925309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573322.373419:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573322.949335:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573323.397446:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573323.973308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573324.421360:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573324.997340:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573325.445424:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573325.509385:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573325.509399:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573325.509405:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573325.509418:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4473 00000100:00080000:2.0:1788573325.509433:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573325.509442:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573325.509492:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488633088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573325.509525:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573325.509529:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573325.509531:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573325.509536:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4473 00000100:00080000:2.0:1788573325.509545:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573325.509550:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573325.509563:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573325.509577:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488633088:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788573325.509579:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488633216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573325.509595:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573325.509599:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573325.509601:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573325.509606:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4473 00000100:00080000:2.0:1788573325.509609:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573325.509613:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573325.509618:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573325.509629:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488633344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573325.509639:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573325.509643:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573325.509645:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573325.509649:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4473 00000100:00080000:2.0:1788573325.509652:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573325.509656:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573325.509661:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573325.509662:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573325.509675:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60e00 x1875450488633472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573325.509686:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573325.509690:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573325.509693:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573325.509698:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4473 00000100:00080000:2.0:1788573325.509702:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573325.509707:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573325.509712:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573325.509719:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573325/real 1788573325] req@ffff94d408c61500 x1875450488633088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573325.509723:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63100 x1875450488633600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573325.509772:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488633088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573325.509788:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573325.509793:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573325.509798:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488633088:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573325.509834:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488633216:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573325.509862:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573325.509877:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573325/real 1788573325] req@ffff94d408c62a00 x1875450488633216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573325.509890:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488633216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573325.509899:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573325.509902:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573325.509906:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488633216:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573325.509912:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488633344:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573325.509933:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573325.509945:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573325/real 1788573325] req@ffff94d408c63800 x1875450488633344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573325.509957:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488633344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573325.509966:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573325.509969:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573325.509973:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488633344:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573325.509980:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488633472:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573325.510000:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573325.510009:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573325/real 1788573325] req@ffff94d408c60e00 x1875450488633472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573325.510028:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60e00 x1875450488633472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573325.510036:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573325.510039:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573325.510043:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488633472:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573325.510049:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488633600:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573325.510066:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573325.510075:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573325/real 1788573325] req@ffff94d408c63100 x1875450488633600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573325.510087:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63100 x1875450488633600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573325.510095:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573325.510098:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573325.510102:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488633600:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573326.021255:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573326.469426:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573327.045329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573327.493420:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573328.069323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573328.517450:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573329.093328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573329.541441:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573330.117369:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573330.565429:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573330.629361:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573330.629377:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573330.629383:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573330.629397:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4478 00000100:00080000:3.0:1788573330.629411:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573330.629421:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573330.629473:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488633728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573330.629507:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573330.629512:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573330.629522:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573330.629527:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4478 00000100:00080000:3.0:1788573330.629531:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573330.629536:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573330.629546:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573330.629559:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488633728:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573330.629560:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488633856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573330.629574:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573330.629577:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573330.629578:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573330.629581:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4478 00000100:00080000:3.0:1788573330.629585:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573330.629587:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573330.629590:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573330.629600:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488633984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573330.629617:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573330.629620:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573330.629621:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573330.629624:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4478 00000100:00080000:3.0:1788573330.629627:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573330.629629:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573330.629633:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573330.629639:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573330.629643:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488634112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573330.629651:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573330.629654:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573330.629655:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573330.629658:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4478 00000100:00080000:3.0:1788573330.629662:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573330.629664:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573330.629667:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573330.629674:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488634240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573330.629683:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573330/real 1788573330] req@ffff94d408c56a00 x1875450488633728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573330.629699:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488633728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573330.629710:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573330.629713:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573330.629717:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488633728:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573330.629739:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488633856:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573330.629756:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573330.629764:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573330/real 1788573330] req@ffff94d408c55180 x1875450488633856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573330.629773:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488633856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573330.629779:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573330.629782:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573330.629784:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488633856:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573330.629787:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488633984:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573330.629798:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573330.629804:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573330/real 1788573330] req@ffff94d408c54000 x1875450488633984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573330.629811:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488633984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573330.629817:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573330.629819:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573330.629821:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488633984:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573330.629825:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488634112:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573330.629837:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573330.629843:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573330/real 1788573330] req@ffff94d408c56300 x1875450488634112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573330.629850:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488634112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573330.629856:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573330.629858:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573330.629859:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488634112:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573330.629863:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488634240:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573330.629873:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573330.629878:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573330/real 1788573330] req@ffff94d408c57480 x1875450488634240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573330.629885:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488634240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573330.629891:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573330.629892:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573330.629894:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488634240:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573331.141278:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573331.589316:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573332.165322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573332.613440:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573333.189280:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573333.637439:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573334.213369:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573334.661450:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573335.237361:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573335.685393:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573335.749346:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573335.749360:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573335.749372:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573335.749386:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4484 00000100:00080000:3.0:1788573335.749402:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573335.749412:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573335.749465:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488634368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573335.749498:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573335.749501:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573335.749503:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573335.749506:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4484 00000100:00080000:3.0:1788573335.749509:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573335.749513:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573335.749522:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573335.749534:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488634496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573335.749543:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573335.749546:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573335.749547:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573335.749550:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4484 00000100:00080000:3.0:1788573335.749555:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573335.749556:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488634368:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573335.749559:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573335.749567:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573335.749578:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488634624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573335.749589:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573335.749592:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573335.749593:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573335.749596:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4484 00000100:00080000:3.0:1788573335.749598:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573335.749601:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573335.749604:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573335.749612:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488634752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573335.749618:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573335.749621:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573335.749622:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573335.749625:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4484 00000100:00080000:3.0:1788573335.749627:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573335.749629:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573335.749632:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573335.749638:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573335.749641:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488634880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573335.749686:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573335/real 1788573335] req@ffff94d408c55f80 x1875450488634368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573335.749701:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488634368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573335.749710:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573335.749714:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573335.749717:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488634368:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573335.749740:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488634496:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573335.749757:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573335.749766:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573335/real 1788573335] req@ffff94d408c54700 x1875450488634496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573335.749774:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488634496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573335.749780:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573335.749783:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573335.749785:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488634496:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573335.749788:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488634624:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573335.749799:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573335.749804:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573335/real 1788573335] req@ffff94d408c57480 x1875450488634624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573335.749812:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488634624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573335.749822:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573335.749825:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573335.749827:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488634624:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573335.749830:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488634752:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573335.749842:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573335.749847:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573335/real 1788573335] req@ffff94d408c56300 x1875450488634752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573335.749854:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488634752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573335.749860:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573335.749861:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573335.749863:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488634752:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573335.749866:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488634880:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573335.749876:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573335.749881:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573335/real 1788573335] req@ffff94d408c54000 x1875450488634880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573335.749888:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488634880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573335.749893:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573335.749895:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573335.749898:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488634880:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573336.261375:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573336.709414:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573337.285321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573337.733371:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573338.309319:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573338.757365:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573339.333297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573339.781401:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573340.357358:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573340.805426:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573340.869313:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573340.869322:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573340.869326:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573340.869335:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4489 00000100:00080000:3.0:1788573340.869348:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573340.869355:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573340.869387:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488635008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573340.869410:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573340.869413:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573340.869414:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573340.869417:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4489 00000100:00080000:3.0:1788573340.869419:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573340.869421:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573340.869428:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573340.869436:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488635136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573340.869444:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573340.869446:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573340.869447:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573340.869452:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4489 00000100:00100000:1.0:1788573340.869454:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488635008:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573340.869454:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573340.869461:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573340.869466:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573340.869473:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488635264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573340.869485:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573340.869488:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573340.869489:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573340.869492:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4489 00000100:00080000:3.0:1788573340.869495:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573340.869497:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573340.869500:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573340.869507:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488635392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573340.869513:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788573340.869518:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573340.869518:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573340.869522:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573340.869525:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4489 00000100:00080000:3.0:1788573340.869528:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573340.869531:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573340.869535:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573340.869541:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488635520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573340.869555:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573340/real 1788573340] req@ffff94d408c55180 x1875450488635008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573340.869569:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488635008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573340.869578:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573340.869581:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573340.869584:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488635008:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573340.869603:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488635136:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573340.869617:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573340.869625:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573340/real 1788573340] req@ffff94d408c56a00 x1875450488635136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573340.869633:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488635136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573340.869639:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573340.869641:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573340.869643:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488635136:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573340.869646:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488635264:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573340.869657:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573340.869663:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573340/real 1788573340] req@ffff94d408c55880 x1875450488635264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573340.869670:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488635264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573340.869675:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573340.869677:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573340.869679:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488635264:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573340.869682:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488635392:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573340.869692:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573340.869697:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573340/real 1788573340] req@ffff94d408c55c00 x1875450488635392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573340.869704:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488635392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573340.869710:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573340.869711:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573340.869713:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488635392:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573340.869716:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488635520:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573340.869725:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573340.869730:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573340/real 1788573340] req@ffff94d408c57800 x1875450488635520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573340.869741:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488635520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573340.869746:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573340.869747:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573340.869749:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488635520:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573341.381380:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573341.829395:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573342.405325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573342.853403:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573343.429326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573343.877325:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573344.453299:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573344.901348:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573345.477295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573345.925375:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573345.989343:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573345.989355:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573345.989360:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573345.989371:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4494 00000100:00080000:3.0:1788573345.989383:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573345.989391:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573345.989432:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488635648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573345.989463:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573345.989467:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573345.989469:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573345.989473:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4494 00000100:00080000:3.0:1788573345.989477:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573345.989482:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573345.989491:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573345.989507:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488635776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573345.989516:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488635648:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573345.989523:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573345.989529:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573345.989531:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573345.989537:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4494 00000100:00080000:3.0:1788573345.989542:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573345.989547:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573345.989553:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573345.989568:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488635904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573345.989577:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573345.989583:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573345.989588:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573345.989590:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573345.989596:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4494 00000100:00080000:3.0:1788573345.989601:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573345.989606:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573345.989612:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573345/real 1788573345] req@ffff94d408c57100 x1875450488635648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788573345.989613:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573345.989633:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488636032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573345.989634:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488635648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573345.989661:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573345.989665:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573345.989666:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573345.989673:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488635648:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573345.989674:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573345.989679:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573345.989694:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4494 00000100:00100000:1.0:1788573345.989697:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488635776:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788573345.989700:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573345.989708:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573345.989715:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573345.989717:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573345.989728:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573345/real 1788573345] req@ffff94d408c54380 x1875450488635776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573345.989736:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488636160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573345.989741:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488635776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573345.989760:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573345.989763:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573345.989768:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488635776:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573345.989777:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488635904:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573345.989799:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573345.989812:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573345/real 1788573345] req@ffff94d408c56680 x1875450488635904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573345.989824:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488635904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573345.989830:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573345.989832:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573345.989834:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488635904:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573345.989838:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488636032:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573345.989853:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573345.989859:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573345/real 1788573345] req@ffff94d408c56d80 x1875450488636032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573345.989867:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488636032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573345.989881:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573345.989884:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573345.989887:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488636032:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573345.989897:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488636160:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573345.989909:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573345.989915:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573345/real 1788573345] req@ffff94d408c54a80 x1875450488636160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573345.989923:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488636160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573345.989928:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573345.989930:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573345.989932:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488636160:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573346.501300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573346.949324:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573347.525322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573347.973367:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573348.549313:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573348.997372:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573349.573319:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573350.021378:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573350.597273:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573351.045327:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573351.109356:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573351.109367:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573351.109372:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573351.109383:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4499 00000100:00080000:3.0:1788573351.109396:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573351.109404:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573351.109455:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488636288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573351.109492:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573351.109497:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573351.109499:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573351.109504:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4499 00000100:00080000:3.0:1788573351.109508:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573351.109514:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573351.109526:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573351.109542:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488636288:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573351.109545:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488636416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573351.109561:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573351.109563:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573351.109565:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573351.109569:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4499 00000100:00080000:3.0:1788573351.109572:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573351.109575:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573351.109580:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573351.109593:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488636544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573351.109600:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573351.109602:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573351.109604:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573351.109608:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4499 00000100:00080000:3.0:1788573351.109611:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573351.109614:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573351.109618:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573351.109628:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488636672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573351.109638:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573351.109642:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573351.109643:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573351.109648:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4499 00000100:00080000:3.0:1788573351.109652:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573351.109657:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573351.109662:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573351.109667:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488636416:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573351.109693:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488636800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573351.109713:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488636544:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573351.109763:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488636672:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573351.109786:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488636800:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573351.621310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573352.069360:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573352.645280:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573353.093323:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573353.669413:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573354.117431:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788573354.181511:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:2.0:1788573354.181559:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788573354.181577:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788573354.181621:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788573354.181628:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788573354.181635:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788573354.181644:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:1.0:1788573354.181729:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573351/real 1788573354] req@ffff94d408c56680 x1875450488636800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573354.181771:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488636800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573354.181783:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573354.181789:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573354.181794:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488636800:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573354.181803:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573351/real 1788573354] req@ffff94d408c57b80 x1875450488636416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573354.181818:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488636416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573354.181826:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573354.181830:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573354.181833:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488636416:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573354.181841:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573351/real 1788573354] req@ffff94d408c54a80 x1875450488636544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573354.181855:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488636544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573354.181870:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573354.181873:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573354.181877:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488636544:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573354.181885:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573351/real 1788573354] req@ffff94d408c56d80 x1875450488636672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573354.181897:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488636672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573354.181904:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573354.181908:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573354.181911:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488636672:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573354.181917:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573351/real 1788573354] req@ffff94d408c54e00 x1875450488636288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573354.181936:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488636288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573354.181944:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573354.181947:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573354.181950:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488636288:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788573354.693367:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573355.141436:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573355.717349:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573356.165418:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573356.229412:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573356.229422:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573356.229426:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573356.229435:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4504 00000100:00080000:2.0:1788573356.229445:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573356.229474:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573356.229509:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488636928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573356.229551:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573356.229555:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573356.229557:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573356.229561:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4504 00000100:00080000:2.0:1788573356.229564:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573356.229569:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573356.229579:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573356.229591:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488637056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573356.229604:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573356.229609:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573356.229611:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573356.229615:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4504 00000100:00080000:2.0:1788573356.229619:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573356.229624:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573356.229629:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573356.229646:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488637184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573356.229649:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488636928:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788573356.229676:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573356.229679:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573356.229682:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573356.229688:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4504 00000100:00080000:2.0:1788573356.229693:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573356.229698:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573356.229705:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573356.229719:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63100 x1875450488637312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573356.229730:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573356.229734:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573356.229737:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573356.229741:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4504 00000100:00080000:2.0:1788573356.229746:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573356.229754:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573356.229760:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573356.229776:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573356.229777:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60e00 x1875450488637440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573356.229845:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573356/real 1788573356] req@ffff94d408c63480 x1875450488636928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573356.229869:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488636928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573356.229882:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573356.229887:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573356.229892:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488636928:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573356.229903:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488637056:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573356.229934:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573356.229950:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573356/real 1788573356] req@ffff94d408c61880 x1875450488637056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573356.229963:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488637056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573356.229972:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573356.229975:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573356.229979:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488637056:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573356.230033:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488637184:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573356.230058:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573356.230070:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573356/real 1788573356] req@ffff94d408c61f80 x1875450488637184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573356.230082:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488637184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573356.230091:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573356.230095:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573356.230098:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488637184:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573356.230105:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488637312:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573356.230127:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573356.230139:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573356/real 1788573356] req@ffff94d408c63100 x1875450488637312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573356.230172:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63100 x1875450488637312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573356.230182:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573356.230185:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573356.230188:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488637312:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573356.230195:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488637440:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573356.230302:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573356.230316:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573356/real 1788573356] req@ffff94d408c60e00 x1875450488637440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573356.230329:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60e00 x1875450488637440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573356.230341:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573356.230345:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573356.230348:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488637440:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573356.741300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573357.189361:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573357.765335:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573358.213372:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573358.789366:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573359.237465:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573359.813844:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573360.261415:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573360.837549:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573361.285395:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573361.349323:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573361.349332:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573361.349335:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573361.349344:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4509 00000100:00080000:3.0:1788573361.349353:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573361.349360:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573361.349393:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488637568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573361.349420:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573361.349423:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573361.349424:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573361.349427:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4509 00000100:00080000:3.0:1788573361.349429:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573361.349431:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573361.349438:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573361.349447:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488637696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573361.349454:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573361.349457:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573361.349458:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573361.349461:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4509 00000100:00080000:3.0:1788573361.349463:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573361.349465:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573361.349469:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573361.349470:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488637568:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573361.349480:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488637824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573361.349487:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573361.349489:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573361.349490:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573361.349493:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4509 00000100:00080000:3.0:1788573361.349496:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573361.349498:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573361.349501:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573361.349508:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488637952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573361.349515:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573361.349517:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573361.349518:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573361.349520:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4509 00000100:00080000:3.0:1788573361.349522:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573361.349524:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573361.349527:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573361.349529:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573361.349537:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488638080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573361.349562:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573361/real 1788573361] req@ffff94d408c54380 x1875450488637568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573361.349576:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488637568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573361.349585:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573361.349588:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573361.349591:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488637568:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573361.349596:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488637696:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573361.349610:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573361.349618:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573361/real 1788573361] req@ffff94d408c57100 x1875450488637696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573361.349626:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488637696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573361.349632:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573361.349634:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573361.349636:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488637696:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573361.349654:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488637824:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573361.349665:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573361.349671:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573361/real 1788573361] req@ffff94d408c57800 x1875450488637824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573361.349678:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488637824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573361.349684:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573361.349687:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573361.349690:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488637824:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573361.349696:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488637952:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573361.349716:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573361.349728:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573361/real 1788573361] req@ffff94d408c55c00 x1875450488637952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573361.349741:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488637952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573361.349747:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573361.349749:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573361.349751:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488637952:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573361.349755:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488638080:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573361.349769:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573361.349776:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573361/real 1788573361] req@ffff94d408c55880 x1875450488638080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573361.349783:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488638080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573361.349788:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573361.349790:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573361.349792:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488638080:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573361.861547:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573362.309428:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573362.885585:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573363.333349:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573363.909453:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573364.357400:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573364.933297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573365.381325:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573365.957310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573366.405369:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573366.469336:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573366.469358:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573366.469366:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573366.469379:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4514 00000100:00080000:2.0:1788573366.469395:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573366.469405:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573366.469457:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488638208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573366.469500:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573366.469504:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573366.469507:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573366.469514:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4514 00000100:00080000:2.0:1788573366.469519:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573366.469524:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573366.469538:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573366.469542:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488638208:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788573366.469558:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488638336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573366.469582:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573366.469590:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573366.469593:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573366.469599:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4514 00000100:00080000:2.0:1788573366.469606:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788573366.469616:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788573366.469616:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573366.469630:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573366.469650:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488638464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573366.469659:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573366/real 1788573366] req@ffff94d408c63800 x1875450488638208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573366.469662:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573366.469678:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573366.469684:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573366.469696:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4514 00000100:00080000:2.0:1788573366.469701:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573366.469703:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488638208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573366.469710:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573366.469727:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788573366.469730:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573366.469736:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573366.469741:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488638592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573366.469743:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488638208:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788573366.469771:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573366.469776:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573366.469778:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573366.469785:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4514 00000100:00100000:1.0:1788573366.469786:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488638336:192.168.202.125@tcp:38:kworker.0 00000100:00080000:2.0:1788573366.469792:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573366.469801:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573366.469807:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573366.469826:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573366.469827:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488638720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573366.469847:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573366/real 1788573366] req@ffff94d408c62a00 x1875450488638336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573366.469865:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488638336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573366.469875:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573366.469880:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573366.469884:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488638336:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573366.469891:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488638464:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573366.469909:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573366.469918:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573366/real 1788573366] req@ffff94d408c61500 x1875450488638464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573366.469926:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488638464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573366.469936:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573366.469938:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573366.469940:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488638464:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573366.469944:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488638592:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573366.469955:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573366.469961:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573366/real 1788573366] req@ffff94d408c63b80 x1875450488638592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573366.469968:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488638592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573366.469973:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573366.469975:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573366.469977:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488638592:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573366.469988:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488638720:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573366.469998:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573366.470004:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573366/real 1788573366] req@ffff94d408c61c00 x1875450488638720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573366.470012:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488638720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573366.470016:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573366.470018:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573366.470021:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488638720:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573366.981355:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573367.429420:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573368.005303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573368.453349:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573369.029338:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573369.477441:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573370.053266:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573370.501316:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573371.077282:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573371.525360:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573371.589297:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573371.589308:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573371.589313:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573371.589324:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4519 00000100:00080000:2.0:1788573371.589335:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573371.589342:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573371.589379:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488638848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573371.589404:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573371.589406:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573371.589408:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573371.589411:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4519 00000100:00080000:2.0:1788573371.589414:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573371.589417:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573371.589424:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573371.589433:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488638976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573371.589442:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573371.589446:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788573371.589451:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488638848:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788573371.589451:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573371.589457:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4519 00000100:00080000:2.0:1788573371.589463:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573371.589467:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573371.589473:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573371.589489:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488639104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573371.589500:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573371.589502:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573371.589504:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573371.589508:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4519 00000100:00080000:2.0:1788573371.589512:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573371.589518:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:1.0:1788573371.589519:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:2.0:1788573371.589525:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573371.589538:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488639232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573371.589546:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573371.589548:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573371.589549:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573371.589554:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4519 00000100:00100000:1.0:1788573371.589555:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573371/real 1788573371] req@ffff94d408c62d80 x1875450488638848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573371.589556:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573371.589562:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573371.589568:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573371.589578:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488639360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573371.589580:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488638848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573371.589601:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573371.589606:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573371.589612:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488638848:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573371.589635:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488638976:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573371.589654:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573371.589663:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573371/real 1788573371] req@ffff94d408c60000 x1875450488638976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573371.589673:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488638976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573371.589679:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573371.589681:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573371.589683:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488638976:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573371.589687:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488639104:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573371.589698:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573371.589704:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573371/real 1788573371] req@ffff94d408c60a80 x1875450488639104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573371.589712:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488639104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573371.589717:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573371.589719:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573371.589721:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488639104:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573371.589725:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488639232:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573371.589740:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573371.589749:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573371/real 1788573371] req@ffff94d408c62680 x1875450488639232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573371.589761:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488639232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573371.589770:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573371.589774:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573371.589777:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488639232:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573371.589784:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488639360:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573371.589801:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573371.589809:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573371/real 1788573371] req@ffff94d408c61180 x1875450488639360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573371.589821:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488639360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573371.589827:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573371.589829:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573371.589831:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488639360:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573372.101292:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573372.549429:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573373.125290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573373.573346:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573374.149274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573374.597400:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573375.173393:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573375.621412:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573376.197371:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573376.645440:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573376.709355:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573376.709369:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573376.709375:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573376.709389:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4525 00000100:00080000:2.0:1788573376.709402:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573376.709411:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573376.709457:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62300 x1875450488639488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573376.709488:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573376.709491:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573376.709493:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573376.709496:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4525 00000100:00080000:2.0:1788573376.709498:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573376.709501:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573376.709509:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573376.709518:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60380 x1875450488639616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573376.709527:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573376.709530:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573376.709531:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573376.709534:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4525 00000100:00080000:2.0:1788573376.709536:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573376.709538:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573376.709542:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573376.709550:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60700 x1875450488639744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573376.709560:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573376.709562:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573376.709564:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573376.709567:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4525 00000100:00080000:2.0:1788573376.709569:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573376.709577:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573376.709579:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488639488:192.168.202.125@tcp:250:kworker.0 00000008:00100000:2.0:1788573376.709582:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573376.709603:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488639872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573376.709611:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573376.709614:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573376.709615:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573376.709619:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4525 00000100:00080000:2.0:1788573376.709621:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573376.709624:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573376.709627:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573376.709635:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488640000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573376.709658:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573376.709694:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573376/real 1788573376] req@ffff94d408c62300 x1875450488639488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573376.709708:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62300 x1875450488639488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573376.709717:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573376.709720:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573376.709723:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488639488:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573376.709729:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488639616:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573376.709744:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573376.709752:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573376/real 1788573376] req@ffff94d408c60380 x1875450488639616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573376.709760:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60380 x1875450488639616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573376.709766:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573376.709768:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573376.709771:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488639616:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573376.709791:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488639744:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573376.709803:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573376.709809:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573376/real 1788573376] req@ffff94d408c60700 x1875450488639744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573376.709816:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60700 x1875450488639744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573376.709822:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573376.709824:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573376.709826:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488639744:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573376.709830:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488639872:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573376.709840:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573376.709846:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573376/real 1788573376] req@ffff94d408c61180 x1875450488639872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573376.709854:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488639872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573376.709858:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573376.709860:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573376.709863:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488639872:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573376.709866:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488640000:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573376.709877:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573376.709882:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573376/real 1788573376] req@ffff94d408c62680 x1875450488640000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573376.709889:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488640000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573376.709893:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573376.709895:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573376.709897:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488640000:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573377.221372:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573377.669434:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573378.245369:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573378.693463:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573379.269359:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573379.717400:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573380.293348:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573380.741461:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573381.317368:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573381.765427:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573381.829485:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573381.829495:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573381.829500:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573381.829509:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4530 00000100:00080000:1.0:1788573381.829520:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573381.829528:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573381.829565:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488640128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573381.829599:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573381.829604:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573381.829605:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573381.829610:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4530 00000100:00080000:1.0:1788573381.829614:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573381.829618:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573381.829628:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573381.829641:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd880 x1875450488640256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573381.829653:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573381.829655:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488640128:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788573381.829659:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573381.829664:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573381.829668:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4530 00000100:00080000:1.0:1788573381.829671:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573381.829675:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573381.829680:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573381.829695:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488640384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573381.829704:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573381.829707:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573381.829709:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573381.829713:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4530 00000100:00080000:1.0:1788573381.829717:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573381.829721:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:0.0:1788573381.829726:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:1.0:1788573381.829727:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573381.829743:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488640512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573381.829759:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573381.829764:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1788573381.829765:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573381/real 1788573381] req@ffff94d4049be300 x1875450488640128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573381.829767:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573381.829776:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4530 00000100:00080000:1.0:1788573381.829783:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573381.829790:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573381.829796:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573381.829801:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488640128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573381.829812:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488640640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573381.829813:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573381.829821:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573381.829829:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488640128:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573381.829855:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488640256:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573381.829881:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573381.829891:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573381/real 1788573381] req@ffff94d4049bd880 x1875450488640256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573381.829906:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd880 x1875450488640256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573381.829912:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573381.829915:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573381.829917:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488640256:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573381.829921:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488640384:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573381.829936:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573381.829942:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573381/real 1788573381] req@ffff94d4049bdc00 x1875450488640384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573381.829950:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488640384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573381.829957:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573381.829961:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573381.829964:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488640384:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573381.829970:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488640512:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573381.829993:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573381.830005:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573381/real 1788573381] req@ffff94d4049be680 x1875450488640512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573381.830017:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488640512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573381.830022:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573381.830025:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573381.830027:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488640512:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573381.830031:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488640640:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573381.830045:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573381.830052:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573381/real 1788573381] req@ffff94d4049bca80 x1875450488640640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573381.830064:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488640640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573381.830069:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573381.830071:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573381.830073:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488640640:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573382.341362:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573382.789353:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573383.365369:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573383.813448:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573384.389276:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573384.837423:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573385.413322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573385.861320:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573386.437265:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573386.885374:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573386.949324:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573386.949337:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573386.949341:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573386.949352:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4535 00000100:00080000:3.0:1788573386.949363:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573386.949371:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573386.949406:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488640768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573386.949436:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573386.949439:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573386.949440:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573386.949444:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4535 00000100:00080000:3.0:1788573386.949447:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573386.949451:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573386.949460:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573386.949473:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488640896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573386.949483:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488640768:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573386.949483:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573386.949492:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573386.949494:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573386.949498:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4535 00000100:00080000:3.0:1788573386.949502:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573386.949505:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573386.949508:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573386.949519:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488641024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573386.949527:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573386.949529:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573386.949531:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573386.949534:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4535 00000100:00080000:3.0:1788573386.949538:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788573386.949552:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573386.949578:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573386/real 1788573386] req@ffff94d408c56a00 x1875450488640768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573386.949595:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488640768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573386.949605:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573386.949608:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:3.0:1788573386.949611:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573386.949614:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488640768:192.168.202.125@tcp:250:kworker.0 00000008:00100000:3.0:1788573386.949622:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573386.949637:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488640896:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573386.949640:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488641152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573386.949658:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573386.949661:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573386.949664:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788573386.949672:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573386.949672:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4535 00000100:00080000:3.0:1788573386.949680:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573386.949685:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573386.949688:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573386/real 1788573386] req@ffff94d408c55180 x1875450488640896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788573386.949691:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573386.949704:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488640896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573386.949708:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488641280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573386.949720:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573386.949727:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573386.949731:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488640896:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573386.949737:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488641024:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573386.949751:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573386.949766:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573386/real 1788573386] req@ffff94d408c54000 x1875450488641024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573386.949773:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488641024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573386.949780:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573386.949782:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573386.949785:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488641024:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573386.949794:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488641152:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573386.949809:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573386.949821:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573386/real 1788573386] req@ffff94d408c56300 x1875450488641152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573386.949843:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488641152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573386.949853:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573386.949858:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573386.949862:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488641152:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573386.949871:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488641280:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573386.949897:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573386.949913:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573386/real 1788573386] req@ffff94d408c57480 x1875450488641280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573386.949959:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488641280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573386.949978:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573386.949985:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573386.949988:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488641280:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573387.461308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573387.909346:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573388.485310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573388.933386:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573389.509288:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573389.957348:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573390.533302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573390.981386:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573391.557341:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573392.005465:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573392.069331:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573392.069346:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573392.069352:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573392.069367:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4540 00000100:00080000:3.0:1788573392.069383:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573392.069394:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573392.069440:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488641408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573392.069467:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573392.069470:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573392.069471:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573392.069474:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4540 00000100:00080000:3.0:1788573392.069477:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573392.069480:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573392.069490:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573392.069505:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488641536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573392.069518:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573392.069523:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573392.069527:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573392.069529:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488641408:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573392.069534:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4540 00000100:00080000:3.0:1788573392.069550:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573392.069554:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573392.069559:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573392.069576:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488641664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573392.069584:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573392.069587:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573392.069589:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573392.069592:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4540 00000100:00080000:3.0:1788573392.069595:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573392.069598:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573392.069602:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573392.069610:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488641792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573392.069617:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788573392.069622:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573392.069623:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573392.069625:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573392.069630:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4540 00000100:00080000:3.0:1788573392.069632:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573392.069635:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573392.069637:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573392.069645:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488641920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573392.069677:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573392/real 1788573392] req@ffff94d408c54700 x1875450488641408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573392.069708:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488641408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573392.069721:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573392.069726:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573392.069731:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488641408:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573392.069774:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488641536:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573392.069801:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573392.069814:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573392/real 1788573392] req@ffff94d408c55f80 x1875450488641536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573392.069829:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488641536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573392.069838:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573392.069842:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573392.069845:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488641536:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573392.069852:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488641664:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573392.069871:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573392.069883:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573392/real 1788573392] req@ffff94d408c57480 x1875450488641664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573392.069896:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488641664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573392.069912:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573392.069915:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573392.069919:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488641664:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573392.069926:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488641792:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573392.069947:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573392.069958:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573392/real 1788573392] req@ffff94d408c56300 x1875450488641792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573392.069971:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488641792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573392.069980:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573392.069983:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573392.069987:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488641792:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573392.069993:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488641920:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573392.070013:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573392.070024:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573392/real 1788573392] req@ffff94d408c54000 x1875450488641920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573392.070037:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488641920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573392.070046:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573392.070050:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573392.070054:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488641920:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573392.581298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573393.029435:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573393.605387:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573394.053396:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573394.629379:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573395.077349:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573395.653339:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573396.101386:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573396.677324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573397.125392:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573397.189304:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573397.189313:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573397.189316:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573397.189325:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4545 00000100:00080000:2.0:1788573397.189334:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573397.189340:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573397.189376:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488642048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573397.189407:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573397.189411:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573397.189412:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573397.189416:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4545 00000100:00080000:2.0:1788573397.189419:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573397.189422:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573397.189430:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573397.189444:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488642176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573397.189451:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488642048:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788573397.189466:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573397.189469:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573397.189470:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573397.189474:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4545 00000100:00080000:2.0:1788573397.189476:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573397.189478:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573397.189481:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573397.189491:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488642304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573397.189497:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573397.189500:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573397.189501:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573397.189504:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4545 00000100:00080000:2.0:1788573397.189508:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573397.189512:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:1.0:1788573397.189517:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:2.0:1788573397.189518:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573397.189533:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488642432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573397.189545:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573397.189548:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573397/real 1788573397] req@ffff94d408c60a80 x1875450488642048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573397.189551:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573397.189554:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573397.189561:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4545 00000100:00080000:2.0:1788573397.189569:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573397.189576:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573397.189582:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488642048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788573397.189583:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788573397.189600:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1788573397.189602:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488642560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573397.189608:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573397.189625:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488642048:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573397.189661:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488642176:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573397.189687:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573397.189701:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573397/real 1788573397] req@ffff94d408c60000 x1875450488642176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573397.189715:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488642176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573397.189724:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573397.189728:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573397.189732:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488642176:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573397.189741:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488642304:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573397.189761:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573397.189773:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573397/real 1788573397] req@ffff94d408c62d80 x1875450488642304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573397.189786:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488642304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573397.189796:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573397.189799:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573397.189803:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488642304:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573397.189809:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488642432:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573397.189836:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573397.189847:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573397/real 1788573397] req@ffff94d408c61c00 x1875450488642432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573397.189870:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488642432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573397.189879:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573397.189883:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573397.189886:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488642432:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573397.189893:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488642560:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573397.189914:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573397.189925:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573397/real 1788573397] req@ffff94d408c63b80 x1875450488642560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573397.189939:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488642560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573397.189947:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573397.189950:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573397.189954:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488642560:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573397.701334:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573398.149389:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573398.725330:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573399.173381:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573399.749302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573400.197417:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573400.773331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573401.221383:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573401.797334:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573402.245416:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573402.309356:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573402.309366:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573402.309371:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573402.309381:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4550 00000100:00080000:0.0:1788573402.309391:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573402.309399:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573402.309435:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488642688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573402.309462:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573402.309467:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573402.309468:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573402.309472:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4550 00000100:00080000:0.0:1788573402.309475:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573402.309480:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573402.309489:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573402.309503:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488642816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573402.309521:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573402.309526:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788573402.309526:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488642688:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788573402.309529:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573402.309543:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4550 00000100:00080000:0.0:1788573402.309549:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573402.309553:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573402.309559:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573402.309578:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488642944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573402.309598:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573402.309606:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573402.309609:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573402.309615:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4550 00000100:00080000:0.0:1788573402.309620:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573402.309624:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573402.309630:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573402.309632:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573402.309647:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488643072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573402.309656:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573402.309658:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573402.309660:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573402.309664:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4550 00000100:00080000:0.0:1788573402.309668:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573402.309671:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573402.309675:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573402.309682:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488643200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573402.309712:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573402/real 1788573402] req@ffff94d408d3dc00 x1875450488642688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573402.309740:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488642688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573402.309754:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573402.309759:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573402.309763:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488642688:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573402.309797:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488642816:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573402.309827:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573402.309841:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573402/real 1788573402] req@ffff94d408d3c380 x1875450488642816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573402.309880:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488642816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573402.309890:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573402.309894:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573402.309897:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488642816:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573402.309905:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488642944:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573402.309925:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573402.309937:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573402/real 1788573402] req@ffff94d408d3c700 x1875450488642944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573402.309950:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488642944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573402.309959:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573402.309963:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573402.309967:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488642944:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573402.309974:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488643072:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573402.309996:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573402.310008:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573402/real 1788573402] req@ffff94d408d3e300 x1875450488643072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573402.310021:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488643072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573402.310037:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573402.310042:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573402.310045:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488643072:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573402.310053:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488643200:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573402.310076:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573402.310088:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573402/real 1788573402] req@ffff94d408d3d880 x1875450488643200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573402.310101:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488643200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573402.310109:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573402.310113:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573402.310116:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488643200:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573402.821345:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573403.269321:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573403.845276:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573404.293350:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573404.869305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573405.317381:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573405.893271:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573406.341327:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573406.917292:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573407.365345:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573407.429419:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573407.429432:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573407.429436:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573407.429445:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4555 00000100:00080000:1.0:1788573407.429456:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573407.429463:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573407.429498:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488643328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573407.429525:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573407.429528:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573407.429529:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573407.429532:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4555 00000100:00080000:1.0:1788573407.429534:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573407.429537:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573407.429544:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573407.429556:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488643456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573407.429563:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573407.429566:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573407.429567:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573407.429570:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4555 00000100:00080000:1.0:1788573407.429571:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573407.429573:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573407.429576:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573407.429584:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488643584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573407.429589:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488643328:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788573407.429592:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573407.429601:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573407.429608:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573407.429612:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4555 00000100:00080000:1.0:1788573407.429614:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573407.429617:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573407.429620:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573407.429626:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488643712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573407.429634:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573407.429636:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573407.429637:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573407.429640:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4555 00000100:00080000:1.0:1788573407.429642:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573407.429644:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573407.429647:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573407.429654:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488643840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573407.429703:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573407.429760:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573407/real 1788573407] req@ffff94d4049bf100 x1875450488643328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573407.429783:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488643328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573407.429797:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573407.429802:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573407.429807:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488643328:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573407.429818:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488643456:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573407.429847:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573407.429863:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573407/real 1788573407] req@ffff94d4049bd500 x1875450488643456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573407.429878:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488643456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573407.429888:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573407.429893:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573407.429897:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488643456:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573407.429930:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488643584:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573407.429956:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573407.429971:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573407/real 1788573407] req@ffff94d4049bf800 x1875450488643584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573407.429985:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488643584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573407.429995:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573407.429999:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573407.430004:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488643584:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573407.430011:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488643712:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573407.430033:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573407.430047:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573407/real 1788573407] req@ffff94d4049bed80 x1875450488643712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573407.430061:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488643712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573407.430070:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573407.430074:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573407.430077:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488643712:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573407.430084:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488643840:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573407.430105:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573407.430118:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573407/real 1788573407] req@ffff94d4049bc700 x1875450488643840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573407.430132:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488643840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573407.430141:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573407.430144:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573407.430148:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488643840:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573407.941303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573408.389444:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573408.965309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573409.413377:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573409.989330:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573410.437390:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573411.013318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573411.461388:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573412.037357:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573412.485394:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573412.549292:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573412.549301:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573412.549305:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573412.549314:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4560 00000100:00080000:1.0:1788573412.549325:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573412.549334:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573412.549375:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488643968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573412.549397:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573412.549399:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573412.549401:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573412.549404:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4560 00000100:00080000:1.0:1788573412.549407:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573412.549410:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573412.549418:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573412.549432:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488644096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573412.549446:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488643968:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788573412.549453:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573412.549459:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573412.549461:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573412.549465:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4560 00000100:00080000:1.0:1788573412.549469:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573412.549473:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573412.549479:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573412.549505:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488644224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573412.549515:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573412.549519:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573412.549521:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573412.549526:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4560 00000100:00080000:1.0:1788573412.549530:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788573412.549535:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788573412.549540:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573412.549546:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573412.549560:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488644352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573412.549570:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573412.549571:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573412/real 1788573412] req@ffff94d4049bc000 x1875450488643968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573412.549579:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573412.549583:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573412.549591:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4560 00000100:00080000:1.0:1788573412.549595:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573412.549600:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488643968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573412.549601:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573412.549608:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573412.549620:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488644480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573412.549622:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573412.549631:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573412.549638:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488643968:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573412.549666:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488644096:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573412.549692:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573412.549704:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573412/real 1788573412] req@ffff94d4049bfb80 x1875450488644096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573412.549716:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488644096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573412.549730:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573412.549733:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573412.549736:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488644096:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573412.549742:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488644224:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573412.549761:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573412.549771:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573412/real 1788573412] req@ffff94d4049bea00 x1875450488644224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573412.549782:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488644224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573412.549791:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573412.549803:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573412.549805:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488644224:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573412.549813:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488644352:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573412.549830:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573412.549838:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573412/real 1788573412] req@ffff94d4049bf480 x1875450488644352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573412.549845:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488644352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573412.549851:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573412.549853:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573412.549855:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488644352:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573412.549859:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488644480:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573412.549869:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573412.549875:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573412/real 1788573412] req@ffff94d4049bc380 x1875450488644480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573412.549882:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc380 x1875450488644480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573412.549887:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573412.549888:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573412.549890:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488644480:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573413.061331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573413.509395:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573414.085280:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573414.533378:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573415.109267:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573415.557403:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573416.133323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573416.581371:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573417.157375:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573417.605449:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573417.669367:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573417.669378:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573417.669383:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573417.669395:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4565 00000100:00080000:0.0:1788573417.669409:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573417.669417:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573417.669462:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ed80 x1875450488644608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573417.669493:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573417.669497:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573417.669498:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573417.669501:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4565 00000100:00080000:0.0:1788573417.669504:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573417.669508:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573417.669518:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573417.669528:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488644736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573417.669537:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573417.669542:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573417.669544:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573417.669544:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488644608:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788573417.669550:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4565 00000100:00080000:0.0:1788573417.669555:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573417.669558:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573417.669562:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573417.669572:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488644864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573417.669595:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573417.669598:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573417.669600:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573417.669603:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4565 00000100:00080000:0.0:1788573417.669606:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573417.669610:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573417.669615:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573417.669628:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488644992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573417.669639:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573417.669646:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573417.669648:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573417.669651:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4565 00000100:00080000:0.0:1788573417.669655:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573417.669659:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573417.669665:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573417.669678:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488645120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573417.669698:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488644736:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573417.669741:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488644864:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573417.669769:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488644992:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573417.669797:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488645120:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573418.181368:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573418.629408:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573419.205345:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573419.653387:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573420.229352:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573420.677393:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788573420.741455:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:3.0:1788573420.741493:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788573420.741507:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:3.0:1788573420.741536:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:3.0:1788573420.741542:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:3.0:1788573420.741548:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:3.0:1788573420.741556:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:1.0:1788573420.741603:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573417/real 1788573420] req@ffff94d408d3f480 x1875450488644736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573420.741665:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488644736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573420.741687:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573420.741693:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573420.741698:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488644736:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573420.741707:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573417/real 1788573420] req@ffff94d408d3d880 x1875450488644864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573420.741721:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488644864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573420.741730:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573420.741734:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573420.741737:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488644864:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573420.741744:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573417/real 1788573420] req@ffff94d408d3e300 x1875450488644992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573420.741756:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488644992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573420.741771:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573420.741775:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573420.741779:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488644992:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573420.741786:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573417/real 1788573420] req@ffff94d408d3c700 x1875450488645120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573420.741806:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488645120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573420.741813:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573420.741817:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573420.741821:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488645120:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573420.741828:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573417/real 1788573420] req@ffff94d408d3ed80 x1875450488644608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573420.741840:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ed80 x1875450488644608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573420.741848:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573420.741855:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573420.741859:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488644608:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788573421.253311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573421.701364:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573422.277306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573422.725369:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573422.789388:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573422.789404:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573422.789410:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573422.789421:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4571 00000100:00080000:1.0:1788573422.789433:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573422.789442:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573422.789478:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488645248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573422.789502:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573422.789505:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573422.789507:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573422.789511:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4571 00000100:00080000:1.0:1788573422.789514:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573422.789519:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488645248:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788573422.789521:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573422.789535:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573422.789552:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488645376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573422.789564:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573422.789568:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573422.789571:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573422.789575:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4571 00000100:00080000:1.0:1788573422.789579:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573422.789583:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:0.0:1788573422.789590:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:1.0:1788573422.789591:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573422.789607:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488645504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573422.789623:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573422.789626:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573422.789629:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573422.789631:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573422/real 1788573422] req@ffff94d4049bd180 x1875450488645248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573422.789636:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4571 00000100:00080000:1.0:1788573422.789642:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573422.789650:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573422.789658:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573422.789667:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488645248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573422.789687:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488645632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573422.789713:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573422.789714:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573422.789722:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573422.789725:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:1.0:1788573422.789726:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573422.789734:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488645248:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788573422.789735:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4571 00000100:00080000:1.0:1788573422.789743:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573422.789750:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573422.789756:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573422.789769:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488645760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573422.789772:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488645376:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573422.789811:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573422.789829:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573422/real 1788573422] req@ffff94d4049bce00 x1875450488645376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573422.789843:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488645376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573422.789854:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573422.789858:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573422.789862:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488645376:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573422.789870:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488645504:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573422.789896:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573422.789912:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573422/real 1788573422] req@ffff94d4049bc380 x1875450488645504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573422.789928:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc380 x1875450488645504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573422.789945:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573422.789950:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573422.789955:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488645504:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573422.789963:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488645632:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573422.789989:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573422.790002:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573422/real 1788573422] req@ffff94d4049bf480 x1875450488645632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573422.790015:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488645632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573422.790025:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573422.790028:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573422.790032:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488645632:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573422.790052:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488645760:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573422.790076:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573422.790088:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573422/real 1788573422] req@ffff94d4049bea00 x1875450488645760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573422.790101:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488645760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573422.790110:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573422.790114:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573422.790117:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488645760:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573423.301309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573423.749340:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573424.325366:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573424.773402:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573425.349355:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573425.797446:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573426.373717:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573426.821443:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573427.397614:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573427.845464:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573427.909310:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573427.909318:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573427.909322:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573427.909330:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4576 00000100:00080000:1.0:1788573427.909340:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573427.909346:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573427.909379:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488645888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573427.909403:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573427.909405:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573427.909407:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573427.909409:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4576 00000100:00080000:1.0:1788573427.909411:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573427.909414:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573427.909421:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573427.909428:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488646016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573427.909436:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573427.909439:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573427.909440:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573427.909443:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4576 00000100:00080000:1.0:1788573427.909445:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573427.909447:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573427.909449:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573427.909455:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488646144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573427.909464:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488645888:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788573427.909469:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573427.909475:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573427.909476:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573427.909479:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4576 00000100:00080000:1.0:1788573427.909481:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573427.909483:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573427.909486:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573427.909495:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488646272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573427.909501:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573427.909503:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573427.909504:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573427.909506:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4576 00000100:00080000:1.0:1788573427.909533:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573427.909535:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573427.909538:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573427.909545:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488646400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573427.909553:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573427.909596:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573427/real 1788573427] req@ffff94d4049bfb80 x1875450488645888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573427.909617:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488645888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573427.909629:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573427.909633:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573427.909637:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488645888:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573427.909645:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488646016:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573427.909671:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573427.909685:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573427/real 1788573427] req@ffff94d4049bc000 x1875450488646016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573427.909699:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488646016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573427.909718:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573427.909722:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573427.909726:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488646016:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573427.909751:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488646144:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573427.909790:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573427.909803:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573427/real 1788573427] req@ffff94d4049bc700 x1875450488646144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573427.909817:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488646144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573427.909826:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573427.909830:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573427.909834:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488646144:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573427.909840:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488646272:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573427.909875:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573427.909889:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573427/real 1788573427] req@ffff94d4049bed80 x1875450488646272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573427.909902:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488646272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573427.909911:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573427.909915:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573427.909919:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488646272:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573427.909925:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488646400:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573427.909946:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573427.909957:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573427/real 1788573427] req@ffff94d4049bf800 x1875450488646400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573427.909969:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488646400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573427.909976:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573427.909980:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573427.909983:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488646400:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573428.423913:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573428.869375:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573429.445371:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573429.893413:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573430.469502:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573430.917464:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573431.493365:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573431.941363:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573432.517298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573432.965383:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573433.029291:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573433.029303:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573433.029308:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573433.029319:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4581 00000100:00080000:1.0:1788573433.029331:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573433.029338:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573433.029371:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488646528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573433.029393:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573433.029395:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573433.029397:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573433.029399:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4581 00000100:00080000:1.0:1788573433.029402:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573433.029404:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573433.029413:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573433.029422:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488646656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573433.029430:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573433.029433:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573433.029435:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573433.029439:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488646528:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788573433.029439:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4581 00000100:00080000:1.0:1788573433.029443:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573433.029448:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573433.029452:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573433.029460:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488646784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573433.029471:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573433.029473:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573433.029474:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573433.029477:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4581 00000100:00080000:1.0:1788573433.029479:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573433.029481:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573433.029484:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573433.029490:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488646912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573433.029497:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573433.029501:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573433.029502:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573433.029506:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4581 00000100:00080000:1.0:1788573433.029511:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573433.029515:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573433.029521:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573433.029524:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573433.029539:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488647040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573433.029565:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573433/real 1788573433] req@ffff94d4049bd500 x1875450488646528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573433.029581:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488646528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573433.029589:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573433.029592:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573433.029595:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488646528:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573433.029614:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488646656:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573433.029631:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573433.029639:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573433/real 1788573433] req@ffff94d4049bf100 x1875450488646656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573433.029648:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488646656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573433.029653:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573433.029656:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573433.029658:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488646656:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573433.029661:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488646784:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573433.029671:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573433.029677:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573433/real 1788573433] req@ffff94d4049bca80 x1875450488646784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573433.029684:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488646784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573433.029689:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573433.029691:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573433.029693:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488646784:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573433.029696:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488646912:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573433.029706:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573433.029712:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573433/real 1788573433] req@ffff94d4049be680 x1875450488646912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573433.029720:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488646912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573433.029725:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573433.029726:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573433.029728:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488646912:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573433.029731:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488647040:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573433.029741:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573433.029746:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573433/real 1788573433] req@ffff94d4049bdc00 x1875450488647040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573433.029753:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488647040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573433.029758:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573433.029760:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573433.029762:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488647040:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573433.541313:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573433.989373:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573434.565280:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573435.013363:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573435.589275:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573436.037377:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573436.613319:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573437.061352:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573437.637331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573438.085324:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573438.149319:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573438.149335:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573438.149340:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573438.149351:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4586 00000100:00080000:3.0:1788573438.149364:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573438.149372:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573438.149414:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488647168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573438.149455:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573438.149461:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573438.149463:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573438.149468:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4586 00000100:00080000:3.0:1788573438.149473:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573438.149478:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573438.149490:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573438.149517:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488647296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573438.149533:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573438.149541:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788573438.149543:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488647168:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573438.149544:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573438.149555:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4586 00000100:00080000:3.0:1788573438.149560:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573438.149565:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573438.149573:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573438.149594:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488647424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573438.149606:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573438.149611:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573438.149613:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573438.149617:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4586 00000100:00080000:3.0:1788573438.149622:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788573438.149629:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573438.149634:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573438.149642:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573438.149658:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488647552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573438.149669:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573438.149674:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573438.149676:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573438.149683:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4586 00000100:00100000:0.0:1788573438.149687:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573438/real 1788573438] req@ffff94d408c55180 x1875450488647168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573438.149690:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573438.149706:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573438.149714:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573438.149719:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488647168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573438.149734:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488647680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573438.149738:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573438.149747:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573438.149760:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488647168:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573438.149791:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488647296:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573438.149813:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573438.149822:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573438/real 1788573438] req@ffff94d408c56a00 x1875450488647296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573438.149831:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488647296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573438.149836:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573438.149839:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573438.149841:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488647296:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573438.149845:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488647424:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573438.149855:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573438.149861:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573438/real 1788573438] req@ffff94d408c55880 x1875450488647424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573438.149869:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488647424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573438.149874:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573438.149876:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573438.149878:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488647424:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573438.149882:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488647552:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573438.149892:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573438.149898:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573438/real 1788573438] req@ffff94d408c55c00 x1875450488647552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573438.149905:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488647552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573438.149910:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573438.149912:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573438.149913:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488647552:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573438.149917:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488647680:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573438.149926:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573438.149931:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573438/real 1788573438] req@ffff94d408c57800 x1875450488647680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573438.149938:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488647680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573438.149942:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573438.149944:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573438.149946:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488647680:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573438.661322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573439.109369:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573439.685296:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573440.133357:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573440.709380:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573441.157438:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573441.733308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573442.181368:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573442.757304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573443.205320:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573443.269268:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573443.269276:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573443.269280:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573443.269288:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4591 00000100:00080000:3.0:1788573443.269297:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573443.269304:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573443.269335:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488647808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573443.269359:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573443.269363:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573443.269365:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573443.269368:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4591 00000100:00080000:3.0:1788573443.269372:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573443.269377:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573443.269387:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573443.269401:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488647936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573443.269409:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488647808:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573443.269418:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573443.269421:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573443.269423:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573443.269427:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4591 00000100:00080000:3.0:1788573443.269430:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573443.269433:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573443.269438:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573443.269450:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488648064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573443.269457:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573443.269460:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573443.269461:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573443.269464:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4591 00000800:00000100:0.0:1788573443.269468:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573443.269468:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573443.269474:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573443.269477:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573443.269486:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488648192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573443.269493:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573443.269496:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573443.269498:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573443.269503:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4591 00000100:00100000:0.0:1788573443.269504:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573443/real 1788573443] req@ffff94d408c57100 x1875450488647808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573443.269505:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573443.269511:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573443.269518:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573443.269526:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488648320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573443.269531:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488647808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573443.269542:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573443.269545:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573443.269549:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488647808:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573443.269567:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488647936:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573443.269582:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573443.269590:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573443/real 1788573443] req@ffff94d408c54380 x1875450488647936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573443.269599:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488647936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573443.269605:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573443.269607:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573443.269609:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488647936:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573443.269612:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488648064:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573443.269622:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573443.269628:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573443/real 1788573443] req@ffff94d408c54e00 x1875450488648064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573443.269635:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488648064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573443.269645:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573443.269657:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573443.269659:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488648064:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573443.269663:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488648192:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573443.269678:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573443.269689:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573443/real 1788573443] req@ffff94d408c56d80 x1875450488648192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573443.269702:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488648192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573443.269711:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573443.269714:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573443.269718:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488648192:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573443.269724:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488648320:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573443.269741:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573443.269749:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573443/real 1788573443] req@ffff94d408c54a80 x1875450488648320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573443.269757:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488648320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573443.269762:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573443.269764:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573443.269768:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488648320:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573443.781295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573444.229333:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573444.805275:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573445.253312:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573445.829302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573446.277346:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573446.853252:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573447.301343:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573447.877277:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573448.325326:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573448.389303:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573448.389312:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573448.389315:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573448.389323:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4596 00000100:00080000:3.0:1788573448.389332:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573448.389338:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573448.389374:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488648448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573448.389396:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573448.389399:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573448.389400:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573448.389403:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4596 00000100:00080000:3.0:1788573448.389405:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573448.389407:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573448.389414:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573448.389423:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488648576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573448.389431:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573448.389436:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788573448.389437:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488648448:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573448.389438:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573448.389446:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4596 00000100:00080000:3.0:1788573448.389448:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573448.389450:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573448.389453:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573448.389463:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488648704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573448.389470:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573448.389472:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573448.389474:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573448.389478:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4596 00000100:00080000:3.0:1788573448.389481:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573448.389483:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573448.389486:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573448.389494:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488648832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573448.389501:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573448.389507:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573448.389510:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573448.389512:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573448.389516:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4596 00000100:00080000:3.0:1788573448.389522:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573448.389525:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573448.389528:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573448.389538:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573448/real 1788573448] req@ffff94d408c57b80 x1875450488648448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573448.389539:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488648960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573448.389571:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488648448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573448.389580:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573448.389583:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573448.389586:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488648448:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573448.389606:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488648576:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573448.389621:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573448.389629:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573448/real 1788573448] req@ffff94d408c56680 x1875450488648576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573448.389650:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488648576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573448.389659:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573448.389661:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573448.389663:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488648576:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573448.389667:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488648704:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573448.389679:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573448.389687:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573448/real 1788573448] req@ffff94d408c54a80 x1875450488648704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573448.389695:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488648704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573448.389700:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573448.389702:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573448.389704:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488648704:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573448.389708:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488648832:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573448.389718:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573448.389723:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573448/real 1788573448] req@ffff94d408c56d80 x1875450488648832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573448.389731:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488648832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573448.389735:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573448.389737:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573448.389739:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488648832:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573448.389742:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488648960:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573448.389751:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573448.389757:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573448/real 1788573448] req@ffff94d408c54e00 x1875450488648960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573448.389764:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488648960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573448.389770:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573448.389771:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573448.389774:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488648960:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573448.901315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573449.349347:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573449.925277:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573450.373351:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573450.949312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573451.397429:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573451.973294:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573452.421365:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573452.997330:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573453.445389:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573453.509318:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573453.509327:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573453.509330:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573453.509339:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4601 00000100:00080000:1.0:1788573453.509348:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573453.509355:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573453.509386:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd880 x1875450488649088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573453.509408:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573453.509410:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573453.509412:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573453.509414:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4601 00000100:00080000:1.0:1788573453.509416:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573453.509419:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573453.509426:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573453.509434:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488649216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573453.509447:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573453.509452:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788573453.509453:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488649088:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788573453.509454:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573453.509462:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4601 00000100:00080000:1.0:1788573453.509465:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573453.509467:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573453.509470:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573453.509480:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488649344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573453.509486:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573453.509489:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573453.509490:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573453.509493:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4601 00000100:00080000:1.0:1788573453.509499:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573453.509502:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573453.509505:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573453.509512:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573453.509514:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488649472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573453.509522:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573453.509524:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573453.509526:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573453.509529:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4601 00000100:00080000:1.0:1788573453.509534:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573453.509538:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573453.509543:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573453.509548:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573453/real 1788573453] req@ffff94d4049bd880 x1875450488649088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573453.509559:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488649600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573453.509564:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd880 x1875450488649088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573453.509587:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573453.509591:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573453.509595:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488649088:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573453.509614:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488649216:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573453.509629:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573453.509637:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573453/real 1788573453] req@ffff94d4049be300 x1875450488649216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573453.509645:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488649216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573453.509650:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573453.509653:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573453.509657:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488649216:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573453.509663:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488649344:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573453.509682:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573453.509694:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573453/real 1788573453] req@ffff94d4049bdc00 x1875450488649344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573453.509707:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488649344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573453.509716:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573453.509719:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573453.509723:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488649344:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573453.509729:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488649472:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573453.509745:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573453.509753:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573453/real 1788573453] req@ffff94d4049be680 x1875450488649472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573453.509760:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488649472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573453.509766:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573453.509768:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573453.509769:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488649472:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573453.509773:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488649600:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573453.509782:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573453.509788:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573453/real 1788573453] req@ffff94d4049bca80 x1875450488649600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573453.509796:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488649600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573453.509801:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573453.509802:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573453.509804:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488649600:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573454.021342:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573454.469392:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573455.045338:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573455.493398:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573456.069272:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573456.517360:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573457.093273:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573457.541409:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573458.117341:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573458.565363:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573458.629284:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573458.629293:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573458.629297:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573458.629305:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4606 00000100:00080000:1.0:1788573458.629314:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573458.629332:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573458.629363:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488649728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573458.629390:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573458.629393:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573458.629395:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573458.629400:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4606 00000100:00080000:1.0:1788573458.629403:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573458.629408:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573458.629419:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573458.629436:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488649856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573458.629450:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573458.629454:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573458.629459:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573458.629461:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488649728:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788573458.629465:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4606 00000100:00080000:1.0:1788573458.629477:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573458.629483:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573458.629488:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573458.629500:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488649984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573458.629508:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573458.629511:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573458.629512:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573458.629516:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4606 00000100:00080000:1.0:1788573458.629518:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573458.629520:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573458.629524:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573458.629533:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488650112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573458.629541:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573458.629544:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573458.629545:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573458.629549:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4606 00000100:00080000:1.0:1788573458.629552:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573458.629555:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573458.629559:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573458.629568:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488650240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573458.629638:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573458.629901:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573458/real 1788573458] req@ffff94d4049bf100 x1875450488649728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573458.629928:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488649728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573458.629940:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573458.629945:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573458.629950:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488649728:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573458.629982:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488649856:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573458.630011:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573458.630025:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573458/real 1788573458] req@ffff94d4049bd500 x1875450488649856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573458.630039:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488649856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573458.630049:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573458.630053:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573458.630056:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488649856:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573458.630063:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488649984:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573458.630086:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573458.630098:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573458/real 1788573458] req@ffff94d4049bf800 x1875450488649984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573458.630111:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488649984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573458.630120:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573458.630124:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573458.630128:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488649984:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573458.630134:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488650112:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573458.630169:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573458.630183:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573458/real 1788573458] req@ffff94d4049bed80 x1875450488650112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573458.630196:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488650112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573458.630246:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573458.630252:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573458.630268:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488650112:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573458.630278:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488650240:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573458.630305:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573458.630331:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573458/real 1788573458] req@ffff94d4049bc700 x1875450488650240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573458.630345:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488650240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573458.630354:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573458.630358:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573458.630361:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488650240:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573459.141314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573459.589394:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573460.165297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573460.613331:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573461.189333:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573461.637390:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573462.213297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573462.661368:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573463.237308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573463.685378:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573463.749342:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573463.749350:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573463.749353:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573463.749362:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4612 00000100:00080000:0.0:1788573463.749371:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573463.749378:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573463.749409:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488650368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573463.749432:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573463.749435:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573463.749437:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573463.749439:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4612 00000100:00080000:0.0:1788573463.749442:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573463.749444:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573463.749451:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573463.749459:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488650496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573463.749478:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573463.749484:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788573463.749484:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488650368:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788573463.749489:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573463.749497:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4612 00000100:00080000:0.0:1788573463.749501:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573463.749506:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573463.749511:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573463.749525:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488650624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573463.749533:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573463.749535:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573463.749537:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573463.749541:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4612 00000100:00080000:0.0:1788573463.749544:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573463.749547:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573463.749551:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788573463.749554:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573463.749561:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488650752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573463.749569:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573463.749571:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573463.749573:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573463.749576:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4612 00000100:00080000:0.0:1788573463.749579:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573463.749581:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573463.749584:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573463.749591:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488650880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573463.749598:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573463/real 1788573463] req@ffff94d408d3c380 x1875450488650368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573463.749622:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488650368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573463.749633:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573463.749638:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573463.749642:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488650368:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573463.749666:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488650496:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573463.749690:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573463.749702:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573463/real 1788573463] req@ffff94d408d3dc00 x1875450488650496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573463.749714:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488650496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573463.749722:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573463.749725:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573463.749729:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488650496:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573463.749735:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488650624:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573463.749752:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573463.749762:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573463/real 1788573463] req@ffff94d408d3c000 x1875450488650624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573463.749772:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488650624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573463.749780:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573463.749784:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573463.749787:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488650624:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573463.749792:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488650752:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573463.749808:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573463.749816:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573463/real 1788573463] req@ffff94d408d3d180 x1875450488650752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573463.749833:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488650752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573463.749841:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573463.749844:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573463.749847:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488650752:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573463.749852:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488650880:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573463.749867:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573463.749876:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573463/real 1788573463] req@ffff94d408d3df80 x1875450488650880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573463.749888:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488650880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573463.749896:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573463.749900:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573463.749903:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488650880:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573464.261269:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573464.709356:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573465.285325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573465.733378:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573466.309284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573466.757427:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573467.333349:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573467.781394:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573468.357281:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573468.805418:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573468.869337:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573468.869348:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573468.869353:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573468.869363:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4617 00000100:00080000:2.0:1788573468.869373:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573468.869381:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573468.869430:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488651008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573468.869460:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573468.869464:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573468.869467:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573468.869471:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4617 00000100:00080000:2.0:1788573468.869475:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573468.869480:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573468.869489:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573468.869506:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488651136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573468.869519:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488651008:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788573468.869520:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573468.869528:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573468.869531:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573468.869535:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4617 00000100:00080000:2.0:1788573468.869539:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573468.869544:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573468.869550:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573468.869565:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488651264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573468.869576:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573468.869580:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573468.869582:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573468.869586:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4617 00000100:00080000:2.0:1788573468.869591:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573468.869599:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:3.0:1788573468.869601:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:2.0:1788573468.869605:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573468.869620:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60e00 x1875450488651392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573468.869631:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573468.869635:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573468.869637:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573468.869644:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4617 00000100:00100000:3.0:1788573468.869645:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573468/real 1788573468] req@ffff94d408c61500 x1875450488651008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573468.869649:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573468.869657:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573468.869665:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573468.869673:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488651008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573468.869680:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63100 x1875450488651520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573468.869694:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573468.869702:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573468.869708:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488651008:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573468.869734:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488651136:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573468.869761:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573468.869776:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573468/real 1788573468] req@ffff94d408c62a00 x1875450488651136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573468.869790:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488651136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573468.869800:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573468.869804:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573468.869807:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488651136:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573468.869813:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488651264:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573468.869835:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573468.869847:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573468/real 1788573468] req@ffff94d408c63800 x1875450488651264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573468.869860:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488651264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573468.869870:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573468.869873:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573468.869877:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488651264:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573468.869884:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488651392:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573468.869904:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573468.869916:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573468/real 1788573468] req@ffff94d408c60e00 x1875450488651392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573468.869928:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60e00 x1875450488651392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573468.869936:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573468.869940:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573468.869943:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488651392:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573468.869949:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488651520:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573468.869977:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573468.869989:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573468/real 1788573468] req@ffff94d408c63100 x1875450488651520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573468.870002:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63100 x1875450488651520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573468.870010:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573468.870013:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573468.870017:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488651520:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573469.381294:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573469.829402:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573470.405309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573470.853392:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573471.429274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573471.877357:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573472.453334:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573472.901394:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573473.477285:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573473.925380:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573473.989330:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573473.989340:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573473.989344:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573473.989355:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4622 00000100:00080000:3.0:1788573473.989366:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573473.989373:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573473.989410:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488651648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573473.989440:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573473.989445:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573473.989447:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573473.989451:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4622 00000100:00080000:3.0:1788573473.989455:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573473.989460:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573473.989470:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573473.989479:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488651648:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573473.989485:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488651776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573473.989499:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573473.989503:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573473.989505:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573473.989509:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4622 00000100:00080000:3.0:1788573473.989513:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573473.989517:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573473.989523:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573473.989535:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488651904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573473.989538:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573473.989558:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573473.989562:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573473.989563:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573473.989568:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4622 00000100:00080000:3.0:1788573473.989572:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573473.989576:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573473.989581:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573473.989586:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573473/real 1788573473] req@ffff94d408c54380 x1875450488651648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573473.989593:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488652032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573473.989605:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488651648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573473.989606:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573473.989616:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573473.989620:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573473.989629:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4622 00000100:00080000:0.0:1788573473.989631:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573473.989633:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573473.989641:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573473.989648:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488651648:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573473.989650:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573473.989658:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573473.989672:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488652160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573473.989679:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488651776:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573473.989714:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573473.989729:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573473/real 1788573473] req@ffff94d408c57100 x1875450488651776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573473.989742:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488651776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573473.989751:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573473.989755:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573473.989759:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488651776:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573473.989766:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488651904:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573473.989786:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573473.989798:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573473/real 1788573473] req@ffff94d408c57800 x1875450488651904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573473.989811:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488651904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573473.989820:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573473.989824:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573473.989827:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488651904:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573473.989834:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488652032:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573473.989854:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573473.989865:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573473/real 1788573473] req@ffff94d408c55c00 x1875450488652032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573473.989878:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488652032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573473.989885:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573473.989889:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573473.989892:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488652032:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573473.989910:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488652160:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573473.989932:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573473.989945:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573473/real 1788573473] req@ffff94d408c55880 x1875450488652160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573473.989959:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488652160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573473.989968:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573473.989971:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573473.989974:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488652160:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573474.501393:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573474.949513:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573475.525377:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573475.973377:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573476.549342:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573476.997445:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573477.573332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573478.021354:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573478.597356:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573479.045429:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573479.109330:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573479.109341:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573479.109346:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573479.109363:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4627 00000100:00080000:3.0:1788573479.109376:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573479.109384:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573479.109428:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488652288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573479.109463:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573479.109468:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573479.109470:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573479.109475:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4627 00000100:00080000:3.0:1788573479.109479:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573479.109484:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573479.109495:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573479.109511:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488652416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573479.109524:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488652288:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573479.109526:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573479.109535:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573479.109538:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573479.109543:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4627 00000100:00080000:3.0:1788573479.109547:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573479.109552:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573479.109558:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573479.109577:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488652544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573479.109589:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573479.109593:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573479.109595:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573479.109599:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4627 00000100:00080000:3.0:1788573479.109604:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573479.109608:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573479.109613:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573479.109622:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488652672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573479.109629:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573479.109636:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573479.109639:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573479.109640:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573479.109644:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4627 00000100:00080000:3.0:1788573479.109646:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573479.109649:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573479.109652:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573479.109660:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488652800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573479.109686:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573479/real 1788573479] req@ffff94d408c56a00 x1875450488652288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573479.109709:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488652288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573479.109722:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573479.109727:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573479.109743:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488652288:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573479.109774:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488652416:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573479.109798:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573479.109808:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573479/real 1788573479] req@ffff94d408c55180 x1875450488652416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573479.109817:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488652416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573479.109828:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573479.109831:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573479.109833:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488652416:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573479.109837:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488652544:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573479.109849:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573479.109855:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573479/real 1788573479] req@ffff94d408c54000 x1875450488652544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573479.109872:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488652544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573479.109878:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573479.109880:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573479.109882:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488652544:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573479.109886:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488652672:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573479.109899:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573479.109906:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573479/real 1788573479] req@ffff94d408c56300 x1875450488652672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573479.109913:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488652672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573479.109918:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573479.109920:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573479.109928:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488652672:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573479.109932:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488652800:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573479.109944:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573479.109951:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573479/real 1788573479] req@ffff94d408c57480 x1875450488652800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573479.109959:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488652800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573479.109964:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573479.109966:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573479.109968:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488652800:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573479.621336:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573480.069380:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573480.645355:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573481.093473:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573481.669393:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573482.117422:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573482.693382:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573483.141429:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573483.717315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573484.165364:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573484.229258:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573484.229267:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573484.229281:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573484.229291:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4632 00000100:00080000:3.0:1788573484.229301:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573484.229307:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573484.229344:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488652928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573484.229368:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573484.229371:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573484.229373:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573484.229376:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4632 00000100:00080000:3.0:1788573484.229378:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573484.229380:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573484.229387:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573484.229396:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488653056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573484.229403:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573484.229406:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573484.229407:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573484.229410:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4632 00000100:00080000:3.0:1788573484.229412:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573484.229417:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488652928:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573484.229418:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573484.229424:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573484.229435:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488653184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573484.229442:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573484.229444:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573484.229446:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573484.229449:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4632 00000100:00080000:3.0:1788573484.229452:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573484.229455:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573484.229458:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573484.229465:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488653312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573484.229472:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573484.229474:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573484.229476:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573484.229478:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4632 00000100:00080000:3.0:1788573484.229480:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573484.229483:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573484.229486:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573484.229494:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488653440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573484.229500:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488653056:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573484.229532:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488653184:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573484.229550:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488653312:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573484.229572:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488653440:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573484.741356:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573485.189381:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573485.765300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573486.213355:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573486.789294:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573487.237365:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788573487.301480:0:2392:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:1.0:1788573487.301516:0:2392:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788573487.301531:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788573487.301568:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788573487.301580:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788573487.301593:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788573487.301606:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:3.0:1788573487.301633:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573484/real 1788573487] req@ffff94d408c57480 x1875450488653184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573487.301664:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488653184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573487.301686:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573487.301692:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573487.301698:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488653184:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573487.301707:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573484/real 1788573487] req@ffff94d408c56300 x1875450488653312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573487.301722:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488653312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573487.301730:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573487.301733:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573487.301737:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488653312:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573487.301744:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573484/real 1788573487] req@ffff94d408c54000 x1875450488653440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573487.301758:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488653440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573487.301767:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573487.301770:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573487.301774:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488653440:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573487.301782:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573484/real 1788573487] req@ffff94d408c55f80 x1875450488652928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573487.301795:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488652928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573487.301804:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573487.301807:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573487.301811:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488652928:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573487.301819:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573484/real 1788573487] req@ffff94d408c54700 x1875450488653056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573487.301828:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488653056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573487.301834:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573487.301838:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573487.301842:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488653056:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788573487.813355:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573488.261407:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573488.837311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573489.285402:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573489.349401:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573489.349412:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573489.349416:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573489.349427:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4637 00000100:00080000:0.0:1788573489.349437:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573489.349445:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573489.349478:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488653568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573489.349508:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573489.349513:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573489.349515:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573489.349519:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4637 00000100:00080000:0.0:1788573489.349523:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573489.349528:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573489.349538:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573489.349554:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488653696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573489.349562:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488653568:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788573489.349573:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573489.349578:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573489.349580:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573489.349585:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4637 00000100:00080000:0.0:1788573489.349589:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573489.349594:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573489.349599:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573489.349613:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488653824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573489.349624:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573489.349628:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573489.349630:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573489.349634:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4637 00000800:00000100:3.0:1788573489.349640:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788573489.349646:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573489.349654:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573489.349662:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573489.349674:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488653952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573489.349685:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573489.349689:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573489.349691:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573489.349695:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573489/real 1788573489] req@ffff94d408d3ce00 x1875450488653568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573489.349698:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4637 00000100:00080000:0.0:1788573489.349705:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573489.349712:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573489.349719:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573489.349729:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488653568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573489.349734:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488654080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573489.349750:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573489.349757:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573489.349762:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488653568:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573489.349789:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488653696:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573489.349824:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573489.349839:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573489/real 1788573489] req@ffff94d408d3ea00 x1875450488653696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573489.349859:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488653696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573489.349883:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573489.349888:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573489.349892:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488653696:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573489.349899:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488653824:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573489.349923:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573489.349935:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573489/real 1788573489] req@ffff94d408d3e680 x1875450488653824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573489.349949:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488653824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573489.349959:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573489.349962:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573489.349966:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488653824:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573489.349973:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488653952:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573489.350002:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573489.350014:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573489/real 1788573489] req@ffff94d408d3d500 x1875450488653952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573489.350027:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488653952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573489.350036:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573489.350039:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573489.350043:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488653952:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573489.350049:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488654080:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573489.350071:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573489.350187:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573489/real 1788573489] req@ffff94d408d3ca80 x1875450488654080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573489.350234:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488654080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573489.350246:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573489.350249:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573489.350253:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488654080:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573489.861331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573490.309396:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573490.885312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573491.333406:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573491.909383:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573492.357498:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573492.934390:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573493.381443:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573493.957747:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573494.405430:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573494.469359:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573494.469371:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573494.469375:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573494.469387:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4642 00000100:00080000:0.0:1788573494.469400:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573494.469408:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573494.469462:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f800 x1875450488654208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573494.469496:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573494.469499:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573494.469501:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573494.469504:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4642 00000100:00080000:0.0:1788573494.469506:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573494.469509:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573494.469519:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573494.469528:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f100 x1875450488654336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573494.469536:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573494.469540:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573494.469541:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573494.469544:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4642 00000100:00080000:0.0:1788573494.469546:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573494.469549:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573494.469552:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573494.469559:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488654208:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573494.469564:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488654464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573494.469574:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573494.469576:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573494.469577:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573494.469580:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4642 00000100:00080000:0.0:1788573494.469584:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573494.469587:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573494.469591:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573494.469599:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488654592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573494.469607:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573494.469610:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573494.469611:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573494.469614:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4642 00000100:00080000:0.0:1788573494.469616:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573494.469618:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573494.469621:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573494.469628:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488654720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573494.469638:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573494.469680:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573494/real 1788573494] req@ffff94d408d3f800 x1875450488654208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573494.469697:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f800 x1875450488654208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573494.469707:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573494.469711:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573494.469715:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488654208:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573494.469732:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488654336:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573494.469755:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573494.469767:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573494/real 1788573494] req@ffff94d408d3f100 x1875450488654336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573494.469781:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f100 x1875450488654336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573494.469792:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573494.469796:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573494.469801:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488654336:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573494.469836:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488654464:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573494.469867:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573494.469885:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573494/real 1788573494] req@ffff94d408d3ca80 x1875450488654464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573494.469901:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488654464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573494.469911:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573494.469916:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573494.469920:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488654464:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573494.469930:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488654592:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573494.470030:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573494.470041:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573494/real 1788573494] req@ffff94d408d3d500 x1875450488654592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573494.470050:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488654592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573494.470056:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573494.470059:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573494.470061:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488654592:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573494.470065:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488654720:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573494.470079:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573494.470085:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573494/real 1788573494] req@ffff94d408d3e680 x1875450488654720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573494.470245:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488654720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573494.470268:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573494.470271:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573494.470275:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488654720:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573494.981366:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573495.429432:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573496.005496:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573496.453438:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573497.029736:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573497.477404:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573498.053408:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573498.501414:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573499.077293:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573499.525366:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573499.589294:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573499.589306:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573499.589310:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573499.589321:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4647 00000100:00080000:0.0:1788573499.589332:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573499.589340:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573499.589374:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488654848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573499.589400:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573499.589405:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573499.589407:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573499.589410:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4647 00000100:00080000:0.0:1788573499.589412:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573499.589415:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573499.589423:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573499.589435:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488654976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573499.589435:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488654848:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788573499.589450:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573499.589452:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573499.589453:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573499.589456:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4647 00000100:00080000:0.0:1788573499.589458:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573499.589461:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573499.589464:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573499.589474:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488655104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573499.589481:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573499.589483:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573499.589484:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573499.589487:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4647 00000100:00080000:0.0:1788573499.589490:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573499.589493:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573499.589497:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788573499.589501:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573499.589509:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488655232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573499.589528:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573499.589531:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573499.589533:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573499.589540:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4647 00000100:00100000:3.0:1788573499.589540:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573499/real 1788573499] req@ffff94d408d3ea00 x1875450488654848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573499.589547:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573499.589556:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573499.589564:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573499.589572:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488654848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573499.589586:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488655360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573499.589587:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573499.589596:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573499.589604:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488654848:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573499.589628:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488654976:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573499.589648:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573499.589657:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573499/real 1788573499] req@ffff94d408d3ce00 x1875450488654976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573499.589666:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488654976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573499.589672:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573499.589675:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573499.589677:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488654976:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573499.589681:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488655104:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573499.589691:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573499.589697:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573499/real 1788573499] req@ffff94d408d3df80 x1875450488655104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573499.589705:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488655104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573499.589710:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573499.589712:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573499.589714:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488655104:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573499.589718:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488655232:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573499.589729:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573499.589744:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573499/real 1788573499] req@ffff94d408d3d180 x1875450488655232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573499.589753:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488655232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573499.589759:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573499.589761:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573499.589763:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488655232:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573499.589767:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488655360:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573499.589778:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573499.589784:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573499/real 1788573499] req@ffff94d408d3c000 x1875450488655360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573499.589795:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488655360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573499.589804:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573499.589807:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573499.589810:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488655360:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573500.101285:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573500.549381:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573501.125331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573501.573391:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573502.149293:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573502.597400:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573503.173297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573503.621367:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573504.197282:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573504.645380:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573504.709373:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573504.709382:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573504.709385:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573504.709395:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4653 00000100:00080000:3.0:1788573504.709404:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573504.709410:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573504.709453:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488655488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573504.709475:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573504.709478:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573504.709480:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573504.709482:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4653 00000100:00080000:3.0:1788573504.709485:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573504.709488:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573504.709497:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573504.709509:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488655488:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573504.709509:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488655616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573504.709524:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573504.709527:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573504.709529:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573504.709531:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4653 00000100:00080000:3.0:1788573504.709534:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573504.709536:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573504.709538:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573504.709545:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488655744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573504.709551:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573504.709553:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573504.709554:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573504.709557:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4653 00000100:00080000:3.0:1788573504.709560:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573504.709563:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573504.709567:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573504.709573:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488655872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573504.709579:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573504.709581:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573504.709583:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573504.709586:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4653 00000800:00000100:0.0:1788573504.709591:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573504.709593:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573504.709597:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573504.709601:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573504.709608:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488656000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573504.709627:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573504/real 1788573504] req@ffff94d408c54700 x1875450488655488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573504.709642:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488655488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573504.709650:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573504.709653:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573504.709656:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488655488:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573504.709674:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488655616:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573504.709691:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573504.709700:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573504/real 1788573504] req@ffff94d408c55f80 x1875450488655616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573504.709708:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488655616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573504.709714:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573504.709716:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573504.709718:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488655616:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573504.709730:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488655744:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573504.709743:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573504.709749:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573504/real 1788573504] req@ffff94d408c54000 x1875450488655744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573504.709757:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488655744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573504.709762:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573504.709764:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573504.709766:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488655744:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573504.709770:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488655872:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573504.709780:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573504.709785:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573504/real 1788573504] req@ffff94d408c56300 x1875450488655872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573504.709797:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488655872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573504.709802:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573504.709804:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573504.709806:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488655872:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573504.709809:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488656000:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573504.709826:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573504.709832:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573504/real 1788573504] req@ffff94d408c57480 x1875450488656000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573504.709840:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488656000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573504.709845:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573504.709847:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573504.709849:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488656000:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573505.221285:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573505.669332:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573506.245312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573506.693355:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573507.269267:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573507.717435:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573508.293317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573508.741357:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573509.317302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573509.765404:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573509.829345:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573509.829353:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573509.829356:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573509.829365:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4658 00000100:00080000:0.0:1788573509.829374:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573509.829381:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573509.829415:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488656128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573509.829442:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573509.829445:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573509.829446:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573509.829449:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4658 00000100:00080000:0.0:1788573509.829451:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573509.829453:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573509.829461:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573509.829470:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488656256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573509.829478:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573509.829481:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573509.829482:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573509.829486:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488656128:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788573509.829488:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4658 00000100:00080000:0.0:1788573509.829500:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573509.829503:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573509.829507:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573509.829516:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ed80 x1875450488656384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573509.829523:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573509.829525:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573509.829526:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573509.829529:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4658 00000100:00080000:0.0:1788573509.829532:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573509.829535:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573509.829539:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573509.829548:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488656512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573509.829551:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788573509.829560:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573509.829562:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573509.829564:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573509.829567:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4658 00000100:00080000:0.0:1788573509.829570:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573509.829573:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573509.829576:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573509.829584:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488656640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573509.829590:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573509/real 1788573509] req@ffff94d408d3dc00 x1875450488656128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573509.829607:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488656128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573509.829615:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573509.829618:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573509.829621:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488656128:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573509.829640:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488656256:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573509.829654:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573509.829662:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573509/real 1788573509] req@ffff94d408d3c380 x1875450488656256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573509.829671:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488656256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573509.829677:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573509.829679:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573509.829681:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488656256:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573509.829684:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488656384:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573509.829694:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573509.829699:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573509/real 1788573509] req@ffff94d408d3ed80 x1875450488656384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573509.829706:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ed80 x1875450488656384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573509.829712:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573509.829713:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573509.829715:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488656384:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573509.829718:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488656512:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573509.829729:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573509.829735:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573509/real 1788573509] req@ffff94d408d3c700 x1875450488656512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573509.829743:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488656512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573509.829747:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573509.829749:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573509.829751:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488656512:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573509.829755:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488656640:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573509.829772:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573509.829782:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573509/real 1788573509] req@ffff94d408d3e300 x1875450488656640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573509.829794:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488656640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573509.829803:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573509.829806:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573509.829810:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488656640:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573510.341290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573510.789346:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573511.365332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573511.813421:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573512.389338:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573512.837414:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573513.413335:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573513.861457:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573514.437340:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573514.885458:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573514.949350:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573514.949359:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573514.949363:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573514.949371:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4663 00000100:00080000:3.0:1788573514.949381:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573514.949387:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573514.949422:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488656768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573514.949448:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573514.949451:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573514.949452:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573514.949455:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4663 00000100:00080000:3.0:1788573514.949457:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573514.949460:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573514.949468:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573514.949477:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488656896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573514.949482:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488656768:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573514.949494:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573514.949497:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573514.949499:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573514.949503:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4663 00000100:00080000:3.0:1788573514.949506:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573514.949508:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573514.949511:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573514.949521:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488657024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573514.949527:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573514.949529:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573514.949530:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573514.949534:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4663 00000100:00080000:3.0:1788573514.949538:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573514.949540:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573514.949543:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573514.949545:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573514.949554:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488657152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573514.949562:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573514.949565:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573514.949566:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573514.949569:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4663 00000100:00080000:3.0:1788573514.949572:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573514.949574:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573514.949578:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573514.949581:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573514/real 1788573514] req@ffff94d408c55180 x1875450488656768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573514.949588:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488657280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573514.949604:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488656768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573514.949613:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573514.949615:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573514.949619:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488656768:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573514.949638:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488656896:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573514.949653:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573514.949661:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573514/real 1788573514] req@ffff94d408c56a00 x1875450488656896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573514.949670:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488656896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573514.949675:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573514.949677:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573514.949679:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488656896:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573514.949683:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488657024:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573514.949692:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573514.949698:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573514/real 1788573514] req@ffff94d408c55880 x1875450488657024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573514.949713:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488657024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573514.949719:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573514.949721:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573514.949723:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488657024:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573514.949726:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488657152:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573514.949738:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573514.949744:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573514/real 1788573514] req@ffff94d408c55c00 x1875450488657152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573514.949751:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488657152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573514.949756:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573514.949758:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573514.949760:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488657152:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573514.949764:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488657280:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573514.949773:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573514.949778:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573514/real 1788573514] req@ffff94d408c57800 x1875450488657280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573514.949785:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488657280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573514.949790:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573514.949792:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573514.949794:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488657280:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573515.461381:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573515.909442:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573516.485370:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573516.933394:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573517.509384:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573517.957406:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573518.533333:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573518.981417:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573519.557358:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573520.005438:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573520.069374:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573520.069385:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573520.069389:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573520.069399:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4668 00000100:00080000:3.0:1788573520.069410:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573520.069418:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573520.069455:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488657408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573520.069486:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573520.069492:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573520.069494:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573520.069498:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4668 00000100:00080000:3.0:1788573520.069501:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573520.069504:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573520.069512:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573520.069522:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488657536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573520.069531:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573520.069535:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573520.069537:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573520.069540:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4668 00000100:00080000:3.0:1788573520.069543:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573520.069546:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573520.069550:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573520.069558:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488657664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573520.069568:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573520.069570:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573520.069572:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573520.069575:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4668 00000100:00080000:3.0:1788573520.069578:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573520.069586:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573520.069589:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488657408:192.168.202.125@tcp:250:kworker.0 00000008:00100000:3.0:1788573520.069593:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573520.069610:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488657792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573520.069619:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573520.069623:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573520.069624:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573520.069627:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4668 00000100:00080000:3.0:1788573520.069630:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573520.069632:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573520.069636:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573520.069644:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488657920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573520.069701:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573520.069756:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573520/real 1788573520] req@ffff94d408c57100 x1875450488657408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573520.069781:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488657408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573520.069801:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573520.069807:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573520.069813:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488657408:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573520.069823:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488657536:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573520.069850:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573520.069866:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573520/real 1788573520] req@ffff94d408c54380 x1875450488657536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573520.069881:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488657536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573520.069889:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573520.069893:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573520.069896:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488657536:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573520.069929:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488657664:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573520.069950:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573520.069960:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573520/real 1788573520] req@ffff94d408c54e00 x1875450488657664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573520.069968:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488657664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573520.069975:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573520.069977:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573520.069980:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488657664:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573520.069985:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488657792:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573520.069997:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573520.070004:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573520/real 1788573520] req@ffff94d408c56d80 x1875450488657792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573520.070012:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488657792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573520.070017:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573520.070019:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573520.070020:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488657792:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573520.070024:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488657920:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573520.070042:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573520.070054:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573520/real 1788573520] req@ffff94d408c54a80 x1875450488657920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573520.070066:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488657920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573520.070075:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573520.070079:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573520.070083:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488657920:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573520.581343:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573521.029429:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573521.605309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573522.053447:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573522.629292:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573523.077333:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573523.653329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573524.101415:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573524.677364:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573525.125400:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573525.189384:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573525.189397:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573525.189402:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573525.189414:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4673 00000100:00080000:0.0:1788573525.189427:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573525.189436:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573525.189482:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488658048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573525.189519:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573525.189523:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573525.189524:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573525.189528:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4673 00000100:00080000:0.0:1788573525.189530:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573525.189533:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573525.189542:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573525.189553:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488658176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573525.189560:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488658048:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788573525.189565:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573525.189570:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573525.189571:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573525.189575:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4673 00000100:00080000:0.0:1788573525.189577:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573525.189580:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573525.189583:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573525.189594:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488658304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573525.189606:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573525.189608:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573525.189610:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573525.189613:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4673 00000100:00080000:0.0:1788573525.189615:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573525.189618:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573525.189621:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573525.189628:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488658432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573525.189634:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573525.189639:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1788573525.189641:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788573525.189642:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573525.189647:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4673 00000100:00080000:0.0:1788573525.189650:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573525.189652:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573525.189656:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573525.189664:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ed80 x1875450488658560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573525.189691:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573525/real 1788573525] req@ffff94d408d3d880 x1875450488658048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573525.189717:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488658048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573525.189727:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573525.189730:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573525.189734:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488658048:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573525.189758:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488658176:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573525.189779:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573525.189789:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573525/real 1788573525] req@ffff94d408d3f480 x1875450488658176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573525.189798:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488658176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573525.189804:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573525.189806:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573525.189809:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488658176:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573525.189812:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488658304:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573525.189823:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573525.189829:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573525/real 1788573525] req@ffff94d408d3e300 x1875450488658304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573525.189837:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488658304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573525.189844:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573525.189848:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573525.189852:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488658304:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573525.189858:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488658432:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573525.189882:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573525.189893:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573525/real 1788573525] req@ffff94d408d3c700 x1875450488658432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573525.189911:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488658432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573525.189916:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573525.189919:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573525.189921:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488658432:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573525.189925:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488658560:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573525.189940:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573525.189947:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573525/real 1788573525] req@ffff94d408d3ed80 x1875450488658560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573525.189955:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ed80 x1875450488658560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573525.189960:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573525.189963:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573525.189965:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488658560:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573525.701373:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573526.149363:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573526.725275:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573527.173361:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573527.749321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573528.197370:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573528.773327:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573529.221366:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573529.797295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573530.245376:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573530.309320:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573530.309329:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573530.309333:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573530.309341:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4678 00000100:00080000:3.0:1788573530.309351:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573530.309359:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573530.309390:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488658688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573530.309409:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573530.309413:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573530.309415:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573530.309419:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4678 00000100:00080000:3.0:1788573530.309422:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573530.309425:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573530.309433:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488658688:192.168.202.125@tcp:250:kworker.0 00000008:00100000:3.0:1788573530.309433:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573530.309448:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488658816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573530.309457:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573530.309459:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573530.309460:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573530.309463:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4678 00000100:00080000:3.0:1788573530.309466:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573530.309469:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573530.309472:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573530.309482:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488658944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573530.309488:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573530.309490:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573530.309492:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788573530.309496:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573530.309498:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4678 00000100:00080000:3.0:1788573530.309504:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573530.309508:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573530.309512:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573530.309523:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488659072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573530.309530:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573530.309532:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573530/real 1788573530] req@ffff94d408c56680 x1875450488658688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573530.309536:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573530.309540:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573530.309548:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4678 00000100:00080000:3.0:1788573530.309554:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573530.309559:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573530.309563:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573530.309567:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488658688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573530.309572:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488659200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573530.309584:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573530.309591:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573530.309596:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488658688:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573530.309621:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488658816:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573530.309647:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573530.309665:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573530/real 1788573530] req@ffff94d408c57b80 x1875450488658816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573530.309680:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488658816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573530.309690:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573530.309694:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573530.309698:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488658816:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573530.309705:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488658944:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573530.309726:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573530.309747:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573530/real 1788573530] req@ffff94d408c54a80 x1875450488658944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573530.309758:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488658944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573530.309768:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573530.309770:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573530.309773:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488658944:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573530.309777:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488659072:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573530.309792:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573530.309799:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573530/real 1788573530] req@ffff94d408c56d80 x1875450488659072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573530.309807:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488659072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573530.309812:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573530.309814:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573530.309816:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488659072:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573530.309820:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488659200:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573530.309831:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573530.309836:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573530/real 1788573530] req@ffff94d408c54e00 x1875450488659200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573530.309843:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488659200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573530.309850:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573530.309853:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573530.309857:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488659200:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573530.821333:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573531.269413:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573531.845306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573532.293354:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573532.869326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573533.317392:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573533.893311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573534.341359:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573534.917358:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573535.365437:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573535.429325:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573535.429333:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573535.429336:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573535.429345:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4683 00000100:00080000:3.0:1788573535.429354:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573535.429360:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573535.429392:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488659328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573535.429416:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573535.429422:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573535.429423:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573535.429426:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4683 00000100:00080000:3.0:1788573535.429429:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573535.429431:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573535.429437:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573535.429446:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488659456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573535.429454:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573535.429457:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573535.429458:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573535.429463:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4683 00000100:00100000:0.0:1788573535.429466:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488659328:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573535.429467:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573535.429473:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573535.429478:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573535.429488:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488659584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573535.429495:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573535.429498:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573535.429500:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573535.429503:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4683 00000100:00080000:3.0:1788573535.429506:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573535.429508:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573535.429513:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573535.429519:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488659712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573535.429526:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573535.429528:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573535.429529:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573535.429531:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4683 00000100:00080000:3.0:1788573535.429534:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788573535.429539:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573535.429540:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573535.429545:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573535.429552:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488659840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573535.429578:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573535/real 1788573535] req@ffff94d408c54380 x1875450488659328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573535.429593:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488659328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573535.429600:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573535.429603:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573535.429606:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488659328:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573535.429613:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488659456:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573535.429636:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573535.429650:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573535/real 1788573535] req@ffff94d408c57100 x1875450488659456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573535.429663:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488659456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573535.429671:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573535.429673:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573535.429675:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488659456:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573535.429695:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488659584:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573535.429709:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573535.429717:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573535/real 1788573535] req@ffff94d408c57800 x1875450488659584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573535.429725:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488659584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573535.429730:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573535.429732:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573535.429735:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488659584:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573535.429738:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488659712:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573535.429748:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573535.429754:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573535/real 1788573535] req@ffff94d408c55c00 x1875450488659712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573535.429761:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488659712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573535.429766:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573535.429767:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573535.429769:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488659712:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573535.429772:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488659840:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573535.429781:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573535.429786:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573535/real 1788573535] req@ffff94d408c55880 x1875450488659840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573535.429794:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488659840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573535.429803:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573535.429806:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573535.429809:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488659840:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573535.941328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573536.389395:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573536.965394:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573537.413381:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573537.989325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573538.437401:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573539.013371:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573539.461385:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573540.037336:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573540.485448:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573540.549281:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573540.549290:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573540.549294:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573540.549302:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4688 00000100:00080000:3.0:1788573540.549311:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573540.549318:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573540.549351:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488659968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573540.549375:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573540.549378:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573540.549379:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573540.549382:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4688 00000100:00080000:3.0:1788573540.549384:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573540.549387:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573540.549394:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573540.549401:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488660096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573540.549409:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573540.549411:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573540.549412:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573540.549415:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4688 00000100:00080000:3.0:1788573540.549417:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573540.549420:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573540.549424:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573540.549433:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488660224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573540.549439:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573540.549442:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573540.549443:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573540.549446:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4688 00000100:00080000:3.0:1788573540.549449:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573540.549451:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573540.549454:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573540.549463:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488659968:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573540.549464:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488660352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573540.549493:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573540.549495:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573540.549496:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573540.549499:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4688 00000100:00080000:3.0:1788573540.549502:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573540.549505:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573540.549508:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573540.549515:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488660480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573540.549559:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573540.549608:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573540/real 1788573540] req@ffff94d408c56a00 x1875450488659968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573540.549632:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488659968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573540.549642:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573540.549645:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573540.549649:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488659968:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573540.549655:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488660096:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573540.549676:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573540.549686:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573540/real 1788573540] req@ffff94d408c55180 x1875450488660096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573540.549694:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488660096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573540.549701:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573540.549703:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573540.549705:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488660096:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573540.549710:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488660224:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573540.549721:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573540.549728:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573540/real 1788573540] req@ffff94d408c57480 x1875450488660224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573540.549736:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488660224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573540.549741:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573540.549743:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573540.549745:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488660224:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573540.549771:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488660352:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573540.549783:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573540.549789:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573540/real 1788573540] req@ffff94d408c56300 x1875450488660352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573540.549797:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488660352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573540.549802:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573540.549804:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573540.549805:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488660352:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573540.549809:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488660480:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573540.549819:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573540.549825:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573540/real 1788573540] req@ffff94d408c54000 x1875450488660480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573540.549833:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488660480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573540.549838:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573540.549841:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573540.549844:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488660480:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573541.061355:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573541.509444:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573542.085295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573542.533438:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573543.109345:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573543.557384:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573544.133304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573544.581379:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573545.157353:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573545.605415:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573545.669414:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573545.669424:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573545.669429:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573545.669439:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4693 00000100:00080000:2.0:1788573545.669450:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573545.669458:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573545.669498:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488660608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573545.669533:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573545.669538:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573545.669540:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573545.669545:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4693 00000100:00080000:2.0:1788573545.669548:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573545.669553:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573545.669563:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573545.669577:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488660736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573545.669589:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573545.669593:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573545.669595:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573545.669599:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4693 00000100:00080000:2.0:1788573545.669603:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573545.669609:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573545.669621:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488660608:192.168.202.125@tcp:250:kworker.0 00000008:00100000:2.0:1788573545.669621:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573545.669648:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488660864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573545.669664:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573545.669667:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573545.669669:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573545.669673:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4693 00000100:00080000:2.0:1788573545.669677:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573545.669682:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573545.669689:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573545.669704:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63100 x1875450488660992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573545.669717:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573545.669721:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573545.669723:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573545.669729:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4693 00000800:00000100:0.0:1788573545.669739:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788573545.669743:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573545.669761:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573545.669768:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573545.669785:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60e00 x1875450488661120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573545.669800:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573545/real 1788573545] req@ffff94d408c61f80 x1875450488660608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573545.669827:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488660608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573545.669841:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573545.669847:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573545.669852:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488660608:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573545.669863:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488660736:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573545.669892:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573545.669907:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573545/real 1788573545] req@ffff94d408c61880 x1875450488660736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573545.669921:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488660736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573545.669937:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573545.669941:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573545.669945:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488660736:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573545.669978:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488660864:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573545.670003:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573545.670016:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573545/real 1788573545] req@ffff94d408c63480 x1875450488660864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573545.670030:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488660864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573545.670040:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573545.670043:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573545.670045:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488660864:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573545.670051:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488660992:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573545.670069:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573545.670077:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573545/real 1788573545] req@ffff94d408c63100 x1875450488660992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573545.670086:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63100 x1875450488660992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573545.670091:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573545.670093:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573545.670096:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488660992:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573545.670100:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488661120:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573545.670111:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573545.670117:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573545/real 1788573545] req@ffff94d408c60e00 x1875450488661120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573545.670124:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60e00 x1875450488661120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573545.670129:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573545.670131:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573545.670133:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488661120:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573546.181316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573546.629367:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573547.205329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573547.653405:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573548.229326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573548.677489:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573549.253360:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573549.701435:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573550.277337:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573550.725428:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573550.789321:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573550.789329:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573550.789333:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573550.789341:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4699 00000100:00080000:2.0:1788573550.789351:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573550.789357:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573550.789390:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488661248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573550.789415:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573550.789418:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573550.789419:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573550.789422:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4699 00000100:00080000:2.0:1788573550.789424:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573550.789427:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573550.789435:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573550.789443:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488661376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573550.789450:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573550.789453:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573550.789454:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573550.789457:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4699 00000100:00080000:2.0:1788573550.789459:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573550.789461:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573550.789464:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573550.789473:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488661504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573550.789475:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488661248:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788573550.789487:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573550.789490:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573550.789491:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573550.789493:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4699 00000100:00080000:2.0:1788573550.789494:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573550.789496:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573550.789500:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573550.789508:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488661632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573550.789515:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573550.789517:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573550.789519:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573550.789521:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4699 00000100:00080000:2.0:1788573550.789523:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573550.789526:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573550.789528:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573550.789535:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488661760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573550.789582:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488661376:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573550.789625:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488661504:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573550.789654:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488661632:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573550.789682:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488661760:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573551.301340:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573551.749434:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573552.325354:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573552.773431:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573553.349319:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573553.797401:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788573553.861477:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:1.0:1788573553.861521:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788573553.861538:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788573553.861589:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788573553.861599:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788573553.861610:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788573553.861620:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:0.0:1788573553.861639:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573550/real 1788573553] req@ffff94d408c61500 x1875450488661504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573553.861667:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488661504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573553.861687:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573553.861692:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573553.861696:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488661504:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573553.861704:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573550/real 1788573553] req@ffff94d408c63b80 x1875450488661632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573553.861717:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488661632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573553.861725:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573553.861729:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573553.861732:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488661632:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573553.861740:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573550/real 1788573553] req@ffff94d408c61c00 x1875450488661760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573553.861751:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488661760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573553.861759:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573553.861763:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573553.861766:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488661760:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573553.861772:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573550/real 1788573553] req@ffff94d408c63800 x1875450488661248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573553.861784:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488661248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573553.861793:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573553.861796:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573553.861800:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488661248:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573553.861807:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573550/real 1788573553] req@ffff94d408c62a00 x1875450488661376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573553.861818:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488661376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573553.861826:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573553.861829:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573553.861833:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488661376:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788573554.373306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573554.821378:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573555.397335:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573555.845400:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573555.909333:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573555.909341:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573555.909345:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573555.909353:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4704 00000100:00080000:0.0:1788573555.909363:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573555.909369:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573555.909400:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488661888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573555.909425:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573555.909434:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573555.909436:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573555.909438:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4704 00000100:00080000:0.0:1788573555.909441:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573555.909444:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573555.909452:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573555.909461:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488662016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573555.909469:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573555.909472:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573555.909474:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573555.909478:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488661888:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788573555.909479:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4704 00000100:00080000:0.0:1788573555.909484:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573555.909490:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573555.909493:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573555.909501:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488662144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573555.909508:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573555.909510:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573555.909511:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573555.909514:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4704 00000100:00080000:0.0:1788573555.909516:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573555.909518:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573555.909522:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573555.909532:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488662272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573555.909548:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573555.909552:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1788573555.909555:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788573555.909556:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573555.909564:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4704 00000100:00080000:0.0:1788573555.909569:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573555.909574:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573555.909580:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573555.909592:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488662400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573555.909601:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573555/real 1788573555] req@ffff94d408d3c380 x1875450488661888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573555.909632:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488661888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573555.909646:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573555.909650:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573555.909655:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488661888:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573555.909685:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488662016:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573555.909713:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573555.909728:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573555/real 1788573555] req@ffff94d408d3dc00 x1875450488662016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573555.909745:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488662016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573555.909756:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573555.909759:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573555.909763:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488662016:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573555.909770:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488662144:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573555.909791:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573555.909805:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573555/real 1788573555] req@ffff94d408d3c000 x1875450488662144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573555.909820:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488662144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573555.909828:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573555.909832:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573555.909836:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488662144:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573555.909843:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488662272:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573555.909866:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573555.909879:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573555/real 1788573555] req@ffff94d408d3d180 x1875450488662272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573555.909892:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488662272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573555.909909:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573555.909913:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573555.909917:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488662272:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573555.909936:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488662400:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573555.909958:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573555.909971:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573555/real 1788573555] req@ffff94d408d3df80 x1875450488662400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573555.909985:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488662400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573555.909994:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573555.909998:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573555.910002:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488662400:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573556.421343:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573556.869419:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573557.445324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573557.893393:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573558.469331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573558.917407:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573559.493491:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573559.941391:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573560.517483:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573560.965383:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573561.029346:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573561.029353:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573561.029357:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573561.029365:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4709 00000100:00080000:3.0:1788573561.029378:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573561.029384:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573561.029417:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488662528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573561.029439:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573561.029443:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573561.029444:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573561.029447:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4709 00000100:00080000:3.0:1788573561.029449:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573561.029451:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573561.029459:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573561.029467:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488662656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573561.029476:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573561.029479:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573561.029481:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573561.029484:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488662528:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573561.029485:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4709 00000100:00080000:3.0:1788573561.029490:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573561.029496:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573561.029499:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573561.029508:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488662784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573561.029514:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573561.029517:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573561.029518:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573561.029521:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4709 00000100:00080000:3.0:1788573561.029524:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573561.029526:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573561.029530:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573561.029537:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488662912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573561.029543:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573561.029546:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573561.029547:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573561.029549:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4709 00000100:00080000:3.0:1788573561.029551:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573561.029555:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:0.0:1788573561.029557:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:3.0:1788573561.029559:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573561.029568:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488663040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573561.029612:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573561/real 1788573561] req@ffff94d408c55f80 x1875450488662528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573561.029634:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488662528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573561.029646:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573561.029651:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573561.029655:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488662528:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573561.029678:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488662656:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573561.029703:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573561.029716:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573561/real 1788573561] req@ffff94d408c54700 x1875450488662656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573561.029731:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488662656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573561.029740:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573561.029744:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573561.029748:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488662656:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573561.029755:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488662784:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573561.029775:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573561.029787:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573561/real 1788573561] req@ffff94d408c54000 x1875450488662784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573561.029801:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488662784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573561.029809:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573561.029813:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573561.029816:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488662784:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573561.029823:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488662912:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573561.029845:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573561.029857:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573561/real 1788573561] req@ffff94d408c56300 x1875450488662912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573561.029870:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488662912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573561.029883:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573561.029886:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573561.029889:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488662912:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573561.029894:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488663040:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573561.029910:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573561.029918:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573561/real 1788573561] req@ffff94d408c57480 x1875450488663040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573561.029928:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488663040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573561.029935:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573561.029938:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573561.029941:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488663040:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573561.541398:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573561.989401:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573562.565276:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573563.013432:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573563.589276:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573564.037470:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573564.613296:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573565.061364:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573565.637267:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573566.085471:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573566.149382:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573566.149389:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573566.149393:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573566.149403:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4714 00000100:00080000:0.0:1788573566.149412:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573566.149419:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573566.149449:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488663168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573566.149471:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573566.149475:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573566.149477:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573566.149480:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4714 00000100:00080000:0.0:1788573566.149483:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573566.149485:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573566.149492:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573566.149503:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488663296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573566.149512:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573566.149517:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788573566.149519:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488663168:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788573566.149520:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573566.149528:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4714 00000100:00080000:0.0:1788573566.149534:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573566.149538:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573566.149544:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573566.149559:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488663424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573566.149568:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573566.149571:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573566.149573:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573566.149577:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4714 00000100:00080000:0.0:1788573566.149581:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573566.149585:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:3.0:1788573566.149591:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:0.0:1788573566.149594:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573566.149610:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488663552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573566.149619:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573566.149622:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573566.149624:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573566.149627:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4714 00000100:00080000:0.0:1788573566.149631:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573566.149634:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573566.149638:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573566.149647:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488663680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573566.149688:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573566/real 1788573566] req@ffff94d408d3ce00 x1875450488663168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573566.149710:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488663168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573566.149721:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573566.149726:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573566.149729:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488663168:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573566.149862:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488663296:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573566.149887:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573566.149899:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573566/real 1788573566] req@ffff94d408d3ea00 x1875450488663296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573566.149909:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488663296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573566.149917:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573566.149921:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573566.149925:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488663296:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573566.149933:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488663424:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573566.149947:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573566.149998:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573566/real 1788573566] req@ffff94d408d3e680 x1875450488663424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573566.150007:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488663424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573566.150013:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573566.150015:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573566.150017:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488663424:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573566.150021:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488663552:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573566.150036:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573566.150043:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573566/real 1788573566] req@ffff94d408d3d500 x1875450488663552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573566.150051:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488663552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573566.150057:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573566.150059:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573566.150061:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488663552:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573566.150065:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488663680:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573566.150074:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573566.150079:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573566/real 1788573566] req@ffff94d408d3ca80 x1875450488663680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573566.150087:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488663680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573566.150092:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573566.150094:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573566.150096:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488663680:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573566.661297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573567.109398:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573567.685279:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573568.133381:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573568.709341:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573569.157385:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573569.733314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573570.181447:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573570.757287:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573571.205396:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573571.269353:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573571.269365:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573571.269370:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573571.269380:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4719 00000100:00080000:0.0:1788573571.269393:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573571.269406:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573571.269453:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f100 x1875450488663808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573571.269490:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573571.269495:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573571.269498:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573571.269502:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4719 00000100:00080000:0.0:1788573571.269507:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573571.269511:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573571.269522:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573571.269535:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f800 x1875450488663936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573571.269544:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573571.269547:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573571.269549:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573571.269553:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4719 00000100:00080000:0.0:1788573571.269555:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573571.269557:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573571.269561:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573571.269565:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488663808:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573571.269572:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488664064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573571.269581:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573571.269583:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573571.269585:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573571.269587:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4719 00000100:00080000:0.0:1788573571.269589:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573571.269592:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573571.269595:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573571.269603:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488664192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573571.269614:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573571.269617:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573571.269618:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573571.269620:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4719 00000100:00080000:0.0:1788573571.269623:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573571.269625:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573571.269628:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573571.269635:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488664320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573571.269654:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573571.269702:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573571/real 1788573571] req@ffff94d408d3f100 x1875450488663808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573571.269725:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f100 x1875450488663808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573571.269738:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573571.269743:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573571.269748:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488663808:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573571.269757:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488663936:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573571.269783:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573571.269798:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573571/real 1788573571] req@ffff94d408d3f800 x1875450488663936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573571.269813:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f800 x1875450488663936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573571.269823:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573571.269827:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573571.269830:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488663936:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573571.269860:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488664064:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573571.269882:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573571.269895:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573571/real 1788573571] req@ffff94d408d3ca80 x1875450488664064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573571.269916:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488664064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573571.269925:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573571.269928:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573571.269932:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488664064:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573571.269938:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488664192:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573571.269958:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573571.269968:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573571/real 1788573571] req@ffff94d408d3d500 x1875450488664192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573571.269980:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488664192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573571.269988:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573571.269997:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573571.270001:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488664192:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573571.270006:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488664320:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573571.270025:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573571.270035:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573571/real 1788573571] req@ffff94d408d3e680 x1875450488664320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573571.270047:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488664320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573571.270062:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573571.270065:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573571.270069:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488664320:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573571.781297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573572.229440:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573572.805291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573573.253459:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573573.829277:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573574.277384:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573574.853374:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573575.301455:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573575.877371:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573576.325465:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573576.389309:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573576.389320:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573576.389325:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573576.389337:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4724 00000100:00080000:3.0:1788573576.389348:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573576.389356:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573576.389400:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488664448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573576.389428:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573576.389430:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573576.389432:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573576.389435:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4724 00000100:00080000:3.0:1788573576.389437:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573576.389440:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573576.389449:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573576.389458:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488664576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573576.389468:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573576.389471:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573576.389474:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573576.389475:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488664448:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573576.389478:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4724 00000100:00080000:3.0:1788573576.389484:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573576.389488:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573576.389491:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573576.389499:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488664704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573576.389505:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573576.389508:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573576.389509:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573576.389512:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4724 00000100:00080000:3.0:1788573576.389515:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573576.389517:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573576.389521:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573576.389527:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488664832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573576.389533:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573576.389536:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573576.389537:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573576.389540:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4724 00000100:00080000:3.0:1788573576.389543:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573576.389546:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573576.389549:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573576.389556:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488664960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573576.389564:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573576.389605:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573576/real 1788573576] req@ffff94d408c55180 x1875450488664448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573576.389621:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488664448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573576.389631:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573576.389634:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573576.389637:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488664448:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573576.389660:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488664576:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573576.389677:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573576.389685:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573576/real 1788573576] req@ffff94d408c56a00 x1875450488664576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573576.389694:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488664576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573576.389700:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573576.389703:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573576.389705:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488664576:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573576.389709:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488664704:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573576.389720:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573576.389726:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573576/real 1788573576] req@ffff94d408c55880 x1875450488664704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573576.389734:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488664704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573576.389739:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573576.389741:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573576.389743:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488664704:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573576.389746:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488664832:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573576.389758:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573576.389766:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573576/real 1788573576] req@ffff94d408c55c00 x1875450488664832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573576.389778:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488664832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573576.389803:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573576.389808:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573576.389816:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488664832:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573576.389823:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488664960:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573576.389840:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573576.389849:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573576/real 1788573576] req@ffff94d408c57800 x1875450488664960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573576.389857:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488664960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573576.389862:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573576.389865:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573576.389867:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488664960:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573576.901358:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573577.349395:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573577.925334:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573578.373394:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573578.949389:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573579.397433:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573579.973330:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573580.421492:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573580.997341:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573581.445373:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573581.509346:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573581.509356:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573581.509361:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573581.509371:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4729 00000100:00080000:1.0:1788573581.509383:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573581.509391:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573581.509436:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488665088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573581.509464:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573581.509467:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573581.509469:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573581.509474:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4729 00000100:00080000:1.0:1788573581.509477:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573581.509482:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573581.509493:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573581.509510:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488665216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573581.509523:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573581.509526:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488665088:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788573581.509527:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573581.509529:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573581.509537:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4729 00000100:00080000:1.0:1788573581.509541:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573581.509544:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573581.509549:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573581.509565:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488665344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573581.509577:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573581.509581:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573581.509584:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573581.509588:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4729 00000100:00080000:1.0:1788573581.509595:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573581.509600:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573581.509605:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573581.509615:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573581.509617:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488665472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573581.509627:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573581.509630:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573581.509632:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573581.509636:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4729 00000100:00080000:1.0:1788573581.509639:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573581.509643:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573581.509647:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573581.509655:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488665600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573581.509668:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573581/real 1788573581] req@ffff94d4049bc000 x1875450488665088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573581.509692:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488665088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573581.509705:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573581.509710:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573581.509715:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488665088:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573581.509744:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488665216:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573581.509766:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573581.509778:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573581/real 1788573581] req@ffff94d4049bfb80 x1875450488665216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573581.509804:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488665216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573581.509817:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573581.509821:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573581.509825:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488665216:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573581.509832:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488665344:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573581.509851:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573581.509862:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573581/real 1788573581] req@ffff94d4049bea00 x1875450488665344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573581.509875:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488665344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573581.509885:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573581.509888:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573581.509892:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488665344:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573581.509898:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488665472:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573581.509915:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573581.509924:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573581/real 1788573581] req@ffff94d4049bf480 x1875450488665472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573581.509938:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488665472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573581.509947:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573581.509951:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573581.509954:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488665472:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573581.509960:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488665600:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573581.509975:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573581.509984:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573581/real 1788573581] req@ffff94d4049bc380 x1875450488665600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573581.509997:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc380 x1875450488665600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573581.510006:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573581.510009:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573581.510013:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488665600:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573582.021365:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573582.469418:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573583.045307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573583.493421:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573584.069310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573584.517405:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573585.093320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573585.541386:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573586.117303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573586.565398:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573586.629357:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573586.629373:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573586.629379:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573586.629393:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4734 00000100:00080000:1.0:1788573586.629408:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573586.629419:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573586.629481:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488665728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573586.629520:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573586.629525:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573586.629527:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573586.629531:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4734 00000100:00080000:1.0:1788573586.629535:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573586.629540:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573586.629552:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573586.629571:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488665856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573586.629586:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488665728:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788573586.629586:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573586.629595:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573586.629599:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573586.629604:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4734 00000100:00080000:1.0:1788573586.629608:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573586.629612:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573586.629618:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573586.629633:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488665984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573586.629644:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573586.629648:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573586.629650:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573586.629654:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4734 00000100:00080000:1.0:1788573586.629658:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573586.629664:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573586.629671:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573586.629674:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573586.629689:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488666112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573586.629701:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573586.629705:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573586.629708:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573586.629713:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4734 00000100:00080000:1.0:1788573586.629718:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573586.629721:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573586/real 1788573586] req@ffff94d4049bce00 x1875450488665728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573586.629727:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573586.629744:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488665728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788573586.629747:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788573586.629761:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573586.629767:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573586.629769:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488666240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573586.629772:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488665728:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573586.629804:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488665856:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573586.629829:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573586.629843:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573586/real 1788573586] req@ffff94d4049bd180 x1875450488665856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573586.629858:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488665856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573586.629867:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573586.629871:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573586.629875:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488665856:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573586.629882:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488665984:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573586.629920:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573586.629930:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573586/real 1788573586] req@ffff94d4049bc380 x1875450488665984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573586.629939:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc380 x1875450488665984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573586.629946:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573586.629948:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573586.629950:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488665984:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573586.629956:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488666112:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573586.629970:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573586.629980:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573586/real 1788573586] req@ffff94d4049bf480 x1875450488666112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573586.629993:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488666112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573586.630002:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573586.630006:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573586.630010:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488666112:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573586.630016:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488666240:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573586.630038:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573586.630051:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573586/real 1788573586] req@ffff94d4049bea00 x1875450488666240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573586.630064:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488666240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573586.630073:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573586.630077:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573586.630080:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488666240:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573587.141261:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573587.589382:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573588.165317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573588.613350:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573589.189331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573589.637459:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573590.213298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573590.661380:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573591.237386:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573591.685470:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573591.749317:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573591.749327:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573591.749332:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573591.749342:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4740 00000100:00080000:1.0:1788573591.749355:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573591.749364:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573591.749409:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488666368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573591.749438:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573591.749441:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573591.749443:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573591.749446:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4740 00000100:00080000:1.0:1788573591.749448:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573591.749452:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573591.749462:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573591.749471:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488666496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573591.749479:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573591.749482:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573591.749485:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573591.749487:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488666368:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788573591.749489:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4740 00000100:00080000:1.0:1788573591.749494:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573591.749499:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573591.749502:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573591.749509:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488666624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573591.749515:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573591.749518:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573591.749519:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573591.749522:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4740 00000100:00080000:1.0:1788573591.749524:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573591.749527:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573591.749530:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573591.749541:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488666752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573591.749550:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573591.749555:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573591.749557:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573591.749560:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4740 00000800:00000100:0.0:1788573591.749567:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788573591.749574:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573591.749585:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573591.749591:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573591.749602:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488666880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573591.749622:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573591/real 1788573591] req@ffff94d4049bfb80 x1875450488666368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573591.749643:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488666368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573591.749653:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573591.749656:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573591.749659:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488666368:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573591.749685:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488666496:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573591.749704:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573591.749714:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573591/real 1788573591] req@ffff94d4049bc000 x1875450488666496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573591.749722:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488666496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573591.749728:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573591.749730:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573591.749733:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488666496:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573591.749736:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488666624:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573591.749757:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573591.749765:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573591/real 1788573591] req@ffff94d4049bc700 x1875450488666624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573591.749773:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488666624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573591.749779:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573591.749781:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573591.749783:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488666624:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573591.749787:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488666752:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573591.749798:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573591.749804:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573591/real 1788573591] req@ffff94d4049bed80 x1875450488666752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573591.749812:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488666752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573591.749823:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573591.749826:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573591.749828:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488666752:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573591.749832:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488666880:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573591.749843:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573591.749849:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573591/real 1788573591] req@ffff94d4049bf800 x1875450488666880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573591.749857:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488666880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573591.749867:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573591.749869:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573591.749872:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488666880:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573592.261290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573592.709417:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573593.285370:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573593.733418:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573594.309361:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573594.757439:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573595.333313:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573595.781406:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573596.357351:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573596.805390:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573596.869337:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573596.869347:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573596.869352:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573596.869363:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4745 00000100:00080000:0.0:1788573596.869375:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573596.869384:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573596.869426:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488667008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573596.869457:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573596.869460:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573596.869462:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573596.869468:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4745 00000100:00080000:0.0:1788573596.869471:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573596.869474:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573596.869483:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573596.869492:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488667136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573596.869501:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573596.869504:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573596.869505:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573596.869508:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4745 00000100:00080000:0.0:1788573596.869511:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573596.869517:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573596.869517:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488667008:192.168.202.125@tcp:250:kworker.0 00000008:00100000:0.0:1788573596.869521:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573596.869532:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488667264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573596.869538:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573596.869540:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573596.869541:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573596.869543:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4745 00000100:00080000:0.0:1788573596.869546:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573596.869548:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573596.869551:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573596.869559:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488667392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573596.869564:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573596.869566:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573596.869567:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573596.869570:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4745 00000100:00080000:0.0:1788573596.869573:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573596.869576:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573596.869580:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573596.869587:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488667520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573596.869610:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573596.869652:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573596/real 1788573596] req@ffff94d408d3ea00 x1875450488667008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573596.869677:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488667008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573596.869691:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573596.869696:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573596.869700:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488667008:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573596.869725:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488667136:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573596.869744:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573596.869755:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573596/real 1788573596] req@ffff94d408d3ce00 x1875450488667136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573596.869764:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488667136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573596.869771:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573596.869773:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573596.869776:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488667136:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573596.869780:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488667264:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573596.869796:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573596.869807:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573596/real 1788573596] req@ffff94d408d3df80 x1875450488667264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573596.869821:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488667264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573596.869831:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573596.869835:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573596.869838:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488667264:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573596.869845:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488667392:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573596.869864:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573596.869872:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573596/real 1788573596] req@ffff94d408d3d180 x1875450488667392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573596.869889:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488667392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573596.869896:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573596.869898:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573596.869900:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488667392:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573596.869907:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488667520:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573596.869919:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573596.869926:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573596/real 1788573596] req@ffff94d408d3c000 x1875450488667520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573596.869933:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488667520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573596.869939:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573596.869941:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573596.869943:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488667520:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573597.381391:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573597.829390:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573598.405318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573598.853356:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573599.429357:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573599.877378:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573600.453334:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573600.901327:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573601.477348:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573601.925420:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573601.989337:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573601.989349:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573601.989354:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573601.989373:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4750 00000100:00080000:0.0:1788573601.989385:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573601.989395:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573601.989440:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488667648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573601.989482:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573601.989487:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573601.989489:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573601.989494:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4750 00000100:00080000:0.0:1788573601.989498:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573601.989502:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573601.989513:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573601.989515:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488667648:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573601.989532:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488667776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573601.989546:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573601.989550:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573601.989553:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573601.989559:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4750 00000100:00080000:0.0:1788573601.989588:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573601.989595:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573601.989600:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788573601.989605:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573601.989620:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ed80 x1875450488667904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573601.989630:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573601.989632:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573601.989634:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573601.989638:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4750 00000100:00080000:0.0:1788573601.989641:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573601.989644:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573601.989648:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573601.989652:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573601/real 1788573601] req@ffff94d408d3dc00 x1875450488667648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573601.989671:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488668032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573601.989679:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488667648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573601.989684:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573601.989695:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573601.989703:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573601.989704:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:0.0:1788573601.989707:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573601.989714:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488667648:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788573601.989718:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4750 00000100:00080000:0.0:1788573601.989726:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573601.989731:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573601.989736:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573601.989748:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488668160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573601.989751:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488667776:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573601.989783:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573601.989800:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573601/real 1788573601] req@ffff94d408d3c380 x1875450488667776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573601.989815:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488667776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573601.989833:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573601.989838:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573601.989842:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488667776:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573601.989849:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488667904:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573601.989868:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573601.989877:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573601/real 1788573601] req@ffff94d408d3ed80 x1875450488667904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573601.989890:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ed80 x1875450488667904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573601.989900:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573601.989913:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573601.989918:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488667904:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573601.989926:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488668032:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573601.989949:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573601.989959:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573601/real 1788573601] req@ffff94d408d3c700 x1875450488668032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573601.989976:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488668032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573601.989983:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573601.989985:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573601.989988:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488668032:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573601.990017:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488668160:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573601.990051:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573601.990067:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573601/real 1788573601] req@ffff94d408d3e300 x1875450488668160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573601.990097:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488668160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573601.990103:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573601.990105:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573601.990109:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488668160:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573602.501299:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573602.949349:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573603.525321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573603.973402:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573604.549329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573604.997388:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573605.573304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573606.021385:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573606.597364:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573607.045391:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573607.109303:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573607.109312:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573607.109315:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573607.109323:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4755 00000100:00080000:3.0:1788573607.109332:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573607.109339:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573607.109372:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488668288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573607.109396:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573607.109400:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573607.109401:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573607.109404:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4755 00000100:00080000:3.0:1788573607.109407:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573607.109409:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573607.109416:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573607.109440:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488668416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573607.109448:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573607.109451:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573607.109453:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573607.109457:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4755 00000100:00080000:3.0:1788573607.109460:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573607.109463:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573607.109469:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573607.109479:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488668288:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573607.109483:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488668544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573607.109498:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573607.109502:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573607.109505:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573607.109510:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4755 00000100:00080000:3.0:1788573607.109515:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573607.109520:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573607.109527:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573607.109540:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488668672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573607.109543:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573607.109553:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573607.109556:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573607.109558:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573607.109563:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4755 00000100:00080000:3.0:1788573607.109567:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573607.109571:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573607.109575:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573607.109580:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573607/real 1788573607] req@ffff94d408c57100 x1875450488668288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573607.109586:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488668800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573607.109601:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488668288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573607.109613:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573607.109617:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573607.109620:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488668288:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573607.109626:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488668416:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573607.109642:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573607.109649:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573607/real 1788573607] req@ffff94d408c54380 x1875450488668416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573607.109658:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488668416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573607.109663:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573607.109665:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573607.109667:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488668416:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573607.109686:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488668544:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573607.109696:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573607.109702:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573607/real 1788573607] req@ffff94d408c54e00 x1875450488668544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573607.109714:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488668544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573607.109719:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573607.109721:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573607.109723:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488668544:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573607.109726:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488668672:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573607.109736:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573607.109742:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573607/real 1788573607] req@ffff94d408c56d80 x1875450488668672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573607.109768:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488668672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573607.109777:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573607.109780:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573607.109784:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488668672:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573607.109790:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488668800:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573607.109812:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573607.109823:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573607/real 1788573607] req@ffff94d408c54a80 x1875450488668800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573607.109834:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488668800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573607.109848:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573607.109852:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573607.109855:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488668800:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573607.621284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573608.069430:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573608.645342:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573609.093456:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573609.669269:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573610.117434:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573610.693359:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573611.141359:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573611.717357:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573612.165425:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573612.229295:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573612.229303:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573612.229307:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573612.229316:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4760 00000100:00080000:3.0:1788573612.229325:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573612.229332:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573612.229364:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488668928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573612.229392:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573612.229395:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573612.229396:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573612.229399:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4760 00000100:00080000:3.0:1788573612.229402:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573612.229405:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573612.229413:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573612.229421:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488669056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573612.229428:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573612.229430:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573612.229432:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573612.229434:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4760 00000100:00080000:3.0:1788573612.229435:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573612.229437:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573612.229440:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573612.229447:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488669184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573612.229454:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573612.229457:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573612.229458:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573612.229461:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4760 00000100:00080000:3.0:1788573612.229464:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573612.229469:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573612.229472:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573612.229478:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488669312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573612.229490:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573612.229494:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573612.229496:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573612.229498:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4760 00000100:00080000:3.0:1788573612.229500:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573612.229509:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488668928:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573612.229510:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573612.229522:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573612.229536:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488669440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573612.229603:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573612.229644:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573612/real 1788573612] req@ffff94d408c57b80 x1875450488668928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573612.229661:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488668928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573612.229672:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573612.229676:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573612.229679:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488668928:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573612.229685:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488669056:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573612.229700:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573612.229708:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573612/real 1788573612] req@ffff94d408c56680 x1875450488669056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573612.229716:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488669056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573612.229722:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573612.229725:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573612.229727:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488669056:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573612.229731:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488669184:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573612.229741:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573612.229747:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573612/real 1788573612] req@ffff94d408c54a80 x1875450488669184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573612.229754:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488669184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573612.229760:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573612.229762:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573612.229763:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488669184:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573612.229787:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488669312:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573612.229799:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573612.229804:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573612/real 1788573612] req@ffff94d408c56d80 x1875450488669312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573612.229811:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488669312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573612.229816:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573612.229818:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573612.229821:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488669312:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573612.229825:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488669440:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573612.229835:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573612.229841:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573612/real 1788573612] req@ffff94d408c54e00 x1875450488669440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573612.229848:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488669440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573612.229854:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573612.229855:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573612.229857:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488669440:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573612.741311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573613.189455:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573613.765334:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573614.213420:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573614.789350:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573615.237437:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573615.813329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573616.261406:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573616.837295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573617.285392:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573617.349361:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573617.349369:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573617.349373:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573617.349382:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4765 00000100:00080000:0.0:1788573617.349391:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573617.349399:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573617.349432:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488669568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573617.349460:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573617.349464:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573617.349466:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573617.349471:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4765 00000100:00080000:0.0:1788573617.349475:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573617.349481:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573617.349490:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573617.349506:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488669696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573617.349512:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488669568:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788573617.349526:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573617.349531:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573617.349533:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573617.349538:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4765 00000100:00080000:0.0:1788573617.349542:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573617.349546:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573617.349552:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573617.349567:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488669824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573617.349577:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573617.349581:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573617.349583:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573617.349588:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4765 00000100:00080000:0.0:1788573617.349592:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573617.349597:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573617.349603:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573617.349617:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488669952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573617.349628:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573617.349632:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1788573617.349636:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488669696:192.168.202.125@tcp:38:kworker.0 00000100:00080000:0.0:1788573617.349637:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573617.349646:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4765 00000100:00080000:0.0:1788573617.349650:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573617.349654:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573617.349660:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573617.349674:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ed80 x1875450488670080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573617.349681:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488669824:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573617.349705:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488669952:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573617.349724:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488670080:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573617.861289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573618.309313:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573618.885339:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573619.333376:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573619.909330:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573620.357359:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788573620.421436:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:2.0:1788573620.421470:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788573620.421484:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788573620.421514:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788573620.421520:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788573620.421528:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788573620.421536:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:1.0:1788573620.421575:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573617/real 1788573620] req@ffff94d408d3ed80 x1875450488670080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573620.421604:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ed80 x1875450488670080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573620.421617:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573620.421622:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573620.421627:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488670080:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573620.421636:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573617/real 1788573620] req@ffff94d408d3d880 x1875450488669696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573620.421650:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488669696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573620.421659:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573620.421663:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573620.421667:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488669696:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573620.421674:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573617/real 1788573620] req@ffff94d408d3e300 x1875450488669824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573620.421687:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488669824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573620.421696:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573620.421700:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573620.421704:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488669824:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573620.421711:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573617/real 1788573620] req@ffff94d408d3c700 x1875450488669952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573620.421723:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488669952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573620.421732:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573620.421735:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573620.421739:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488669952:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573620.421745:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573617/real 1788573620] req@ffff94d408d3f480 x1875450488669568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573620.421758:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488669568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573620.421765:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573620.421768:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573620.421772:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488669568:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573620.933292:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573621.381378:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573621.957285:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573622.405329:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573622.469327:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573622.469339:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573622.469344:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573622.469355:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4770 00000100:00080000:3.0:1788573622.469366:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573622.469376:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573622.469410:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488670208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573622.469440:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573622.469444:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573622.469446:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573622.469450:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4770 00000100:00080000:3.0:1788573622.469454:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573622.469459:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573622.469470:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573622.469492:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488670208:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573622.469493:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488670336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573622.469512:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573622.469515:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573622.469518:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573622.469522:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4770 00000100:00080000:3.0:1788573622.469526:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573622.469530:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573622.469536:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573622.469550:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488670464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573622.469560:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573622.469564:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1788573622.469569:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573622.469569:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573622.469578:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4770 00000100:00080000:3.0:1788573622.469589:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573622.469594:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573622.469601:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573622.469610:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573622/real 1788573622] req@ffff94d408c54380 x1875450488670208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573622.469616:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488670592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573622.469639:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573622.469643:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1788573622.469647:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488670208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573622.469648:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573622.469657:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4770 00000100:00080000:3.0:1788573622.469664:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573622.469671:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00080000:1.0:1788573622.469673:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788573622.469676:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788573622.469681:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573622.469689:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488670208:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573622.469693:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488670720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573622.469734:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488670336:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573622.469762:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573622.469808:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573622/real 1788573622] req@ffff94d408c57100 x1875450488670336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573622.469822:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488670336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573622.469830:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573622.469834:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573622.469837:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488670336:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573622.469844:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488670464:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573622.469865:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573622.469878:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573622/real 1788573622] req@ffff94d408c57800 x1875450488670464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573622.469890:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488670464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573622.469904:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573622.469907:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573622.469911:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488670464:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573622.469917:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488670592:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573622.469938:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573622.469949:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573622/real 1788573622] req@ffff94d408c55c00 x1875450488670592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573622.469961:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488670592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573622.469970:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573622.469973:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573622.469977:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488670592:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573622.469983:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488670720:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573622.470004:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573622.470016:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573622/real 1788573622] req@ffff94d408c55880 x1875450488670720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573622.470029:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488670720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573622.470038:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573622.470041:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573622.470045:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488670720:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573622.981278:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573623.429374:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573624.005345:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573624.453444:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573625.029315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573625.477406:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573626.053585:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573626.501436:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573627.077545:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573627.525381:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573627.589291:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573627.589319:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573627.589325:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573627.589341:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4775 00000100:00080000:3.0:1788573627.589356:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573627.589367:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573627.589424:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488670848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573627.589463:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573627.589472:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573627.589474:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573627.589478:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4775 00000100:00080000:3.0:1788573627.589483:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573627.589488:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573627.589500:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573627.589515:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488670848:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573627.589519:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488670976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573627.589533:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573627.589538:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573627.589540:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573627.589545:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4775 00000100:00080000:3.0:1788573627.589549:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573627.589554:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573627.589560:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573627.589577:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488671104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573627.589588:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573627.589592:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573627.589594:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788573627.589598:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573627.589603:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4775 00000100:00080000:3.0:1788573627.589611:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573627.589617:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573627.589624:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573627.589637:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573627/real 1788573627] req@ffff94d408c56a00 x1875450488670848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573627.589642:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488671232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573627.589661:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573627.589664:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488670848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573627.589674:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573627.589679:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:3.0:1788573627.589680:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1788573627.589684:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488670848:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573627.589685:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573627.589696:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4775 00000100:00080000:3.0:1788573627.589714:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573627.589719:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573627.589726:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573627.589728:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488670976:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573627.589747:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488671360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573627.589757:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573627.589773:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573627/real 1788573627] req@ffff94d408c55180 x1875450488670976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573627.589794:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488670976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573627.589801:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573627.589804:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573627.589807:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488670976:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573627.589811:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488671104:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573627.589825:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573627.589833:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573627/real 1788573627] req@ffff94d408c57480 x1875450488671104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573627.589841:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488671104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573627.589847:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573627.589850:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573627.589852:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488671104:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573627.589857:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488671232:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573627.589877:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573627.589889:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573627/real 1788573627] req@ffff94d408c56300 x1875450488671232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573627.589903:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488671232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573627.589912:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573627.589915:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573627.589919:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488671232:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573627.589938:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488671360:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573627.589961:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573627.589973:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573627/real 1788573627] req@ffff94d408c54000 x1875450488671360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573627.589981:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488671360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573627.589992:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573627.589994:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573627.589997:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488671360:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573628.101343:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573628.549414:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573629.125643:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573629.573421:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573630.149355:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573630.597427:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573631.173688:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573631.621327:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573632.197342:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573632.645380:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573632.709285:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573632.709297:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573632.709302:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573632.709313:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4781 00000100:00080000:3.0:1788573632.709327:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573632.709335:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573632.709381:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488671488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573632.709416:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573632.709422:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573632.709424:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573632.709433:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4781 00000100:00080000:3.0:1788573632.709437:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573632.709443:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573632.709454:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573632.709459:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488671488:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573632.709470:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488671616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573632.709483:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573632.709487:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573632.709489:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573632.709493:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4781 00000100:00080000:3.0:1788573632.709496:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573632.709500:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573632.709504:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573632.709517:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488671744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573632.709528:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788573632.709536:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573632.709541:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573632.709546:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573632.709553:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4781 00000100:00080000:3.0:1788573632.709559:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573632.709565:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573632.709572:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573632.709580:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573632/real 1788573632] req@ffff94d408c54700 x1875450488671488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573632.709590:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488671872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573632.709614:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573632.709621:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1788573632.709623:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488671488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573632.709639:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573632.709648:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4781 00000100:00080000:3.0:1788573632.709653:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573632.709663:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573632.709668:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788573632.709670:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573632.709679:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573632.709683:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488672000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573632.709692:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488671488:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573632.709735:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488671616:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573632.709758:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573632.709768:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573632/real 1788573632] req@ffff94d408c55f80 x1875450488671616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573632.709778:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488671616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573632.709784:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573632.709787:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573632.709789:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488671616:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573632.709794:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488671744:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573632.709806:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573632.709812:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573632/real 1788573632] req@ffff94d408c54000 x1875450488671744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573632.709819:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488671744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573632.709825:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573632.709827:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573632.709829:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488671744:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573632.709833:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488671872:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573632.709843:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573632.709849:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573632/real 1788573632] req@ffff94d408c56300 x1875450488671872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573632.709857:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488671872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573632.709861:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573632.709863:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573632.709866:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488671872:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573632.709871:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488672000:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573632.709887:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573632.709899:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573632/real 1788573632] req@ffff94d408c57480 x1875450488672000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573632.709912:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488672000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573632.709921:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573632.709925:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573632.709927:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488672000:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573633.221357:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573633.669385:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573634.245301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573634.693405:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573635.269302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573635.717422:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573636.293357:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573636.741438:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573637.317296:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573637.765422:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573637.829357:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573637.829368:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573637.829374:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573637.829388:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4786 00000100:00080000:3.0:1788573637.829403:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573637.829413:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573637.829458:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488672128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573637.829498:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573637.829503:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573637.829505:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573637.829509:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4786 00000100:00080000:3.0:1788573637.829514:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573637.829519:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573637.829531:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573637.829553:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488672256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573637.829579:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573637.829585:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788573637.829587:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488672128:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573637.829588:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573637.829598:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4786 00000100:00080000:3.0:1788573637.829603:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573637.829609:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573637.829615:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573637.829631:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488672384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573637.829674:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573637.829703:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573637.829707:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1788573637.829709:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573637/real 1788573637] req@ffff94d408c55180 x1875450488672128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573637.829710:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573637.829719:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4786 00000100:00080000:3.0:1788573637.829726:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573637.829732:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573637.829736:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488672128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788573637.829739:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788573637.829752:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788573637.829756:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488672512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573637.829759:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573637.829768:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488672128:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573637.829772:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573637.829777:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573637.829778:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573637.829782:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4786 00000100:00080000:3.0:1788573637.829785:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573637.829787:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573637.829790:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573637.829797:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488672256:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573637.829800:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488672640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573637.829821:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573637.829833:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573637/real 1788573637] req@ffff94d408c56a00 x1875450488672256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573637.829847:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488672256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573637.829857:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573637.829861:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573637.829866:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488672256:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573637.829874:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488672384:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573637.829900:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573637.829911:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573637/real 1788573637] req@ffff94d408c55880 x1875450488672384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573637.829921:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488672384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573637.829928:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573637.829930:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573637.829933:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488672384:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573637.829938:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488672512:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573637.829955:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573637.829962:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573637/real 1788573637] req@ffff94d408c55c00 x1875450488672512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573637.829972:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488672512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573637.829978:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573637.829980:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573637.829982:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488672512:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573637.829998:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488672640:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573637.830011:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573637.830020:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573637/real 1788573637] req@ffff94d408c57800 x1875450488672640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573637.830028:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488672640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573637.830034:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573637.830037:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573637.830039:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488672640:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573638.341317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573638.789418:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573639.365352:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573639.813362:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573640.389329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573640.837449:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573641.413310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573641.861419:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573642.437330:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573642.885381:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573642.949353:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573642.949364:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573642.949369:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573642.949379:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4791 00000100:00080000:2.0:1788573642.949392:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573642.949401:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573642.949466:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488672768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573642.949525:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573642.949528:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573642.949529:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573642.949533:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4791 00000100:00080000:2.0:1788573642.949536:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573642.949540:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573642.949550:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573642.949562:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488672896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573642.949571:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573642.949575:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788573642.949578:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488672768:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788573642.949578:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573642.949586:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4791 00000100:00080000:2.0:1788573642.949589:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573642.949592:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573642.949596:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573642.949605:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488673024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573642.949611:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573642.949614:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573642.949616:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573642.949618:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4791 00000100:00080000:2.0:1788573642.949621:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573642.949623:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573642.949627:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573642.949634:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488673152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573642.949639:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573642.949641:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573642.949642:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573642.949645:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4791 00000100:00080000:2.0:1788573642.949651:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573642.949653:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573642.949656:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573642.949667:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488673280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573642.949669:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573642.949722:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573642/real 1788573642] req@ffff94d408c62d80 x1875450488672768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573642.949746:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488672768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573642.949756:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573642.949760:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573642.949764:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488672768:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573642.949789:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488672896:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573642.949809:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573642.949819:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573642/real 1788573642] req@ffff94d408c60000 x1875450488672896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573642.949828:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488672896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573642.949835:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573642.949838:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573642.949842:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488672896:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573642.949849:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488673024:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573642.949869:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573642.949882:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573642/real 1788573642] req@ffff94d408c60a80 x1875450488673024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573642.949895:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488673024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573642.949911:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573642.949915:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573642.949919:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488673024:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573642.949926:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488673152:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573642.949949:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573642.949976:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573642/real 1788573642] req@ffff94d408c62680 x1875450488673152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573642.949991:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488673152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573642.949999:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573642.950002:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573642.950006:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488673152:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573642.950012:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488673280:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573642.950031:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573642.950040:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573642/real 1788573642] req@ffff94d408c61180 x1875450488673280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573642.950052:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488673280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573642.950060:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573642.950063:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573642.950067:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488673280:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573643.461315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573643.909481:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573644.485277:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573644.933370:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573645.509324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573645.957474:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573646.533310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573646.981318:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573647.557292:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573648.005506:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573648.069319:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573648.069333:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573648.069336:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573648.069361:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4796 00000100:00080000:2.0:1788573648.069382:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573648.069395:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573648.069476:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60700 x1875450488673408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573648.069508:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573648.069512:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573648.069513:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573648.069516:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4796 00000100:00080000:2.0:1788573648.069519:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573648.069522:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573648.069530:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573648.069540:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60380 x1875450488673536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573648.069548:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573648.069552:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573648.069554:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573648.069558:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488673408:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788573648.069559:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4796 00000100:00080000:2.0:1788573648.069565:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573648.069569:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573648.069572:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573648.069581:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62300 x1875450488673664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573648.069588:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573648.069591:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573648.069592:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573648.069594:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4796 00000100:00080000:2.0:1788573648.069596:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573648.069599:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573648.069602:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573648.069610:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488673792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573648.069616:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573648.069618:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573648.069619:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573648.069621:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4796 00000100:00080000:2.0:1788573648.069624:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788573648.069628:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788573648.069629:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573648.069635:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573648.069642:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488673920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573648.069672:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573648/real 1788573648] req@ffff94d408c60700 x1875450488673408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573648.069687:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60700 x1875450488673408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573648.069696:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573648.069698:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573648.069702:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488673408:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573648.069738:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488673536:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573648.069757:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573648.069767:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573648/real 1788573648] req@ffff94d408c60380 x1875450488673536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573648.069786:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60380 x1875450488673536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573648.069792:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573648.069795:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573648.069797:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488673536:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573648.069801:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488673664:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573648.069825:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573648.069832:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573648/real 1788573648] req@ffff94d408c62300 x1875450488673664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573648.069850:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62300 x1875450488673664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573648.069856:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573648.069858:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573648.069860:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488673664:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573648.069864:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488673792:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573648.069875:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573648.069881:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573648/real 1788573648] req@ffff94d408c61180 x1875450488673792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573648.069888:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488673792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573648.069893:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573648.069895:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573648.069896:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488673792:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573648.069900:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488673920:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573648.069920:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573648.069926:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573648/real 1788573648] req@ffff94d408c62680 x1875450488673920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573648.069934:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488673920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573648.069939:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573648.069941:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573648.069943:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488673920:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573648.581307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573649.029383:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573649.605315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573650.053364:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573650.629286:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573651.077319:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573651.653625:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573652.101431:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573652.677310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573653.125337:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573653.189297:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573653.189360:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573653.189364:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573653.189374:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4801 00000100:00080000:0.0:1788573653.189386:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573653.189395:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573653.189435:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488674048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573653.189464:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573653.189468:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573653.189469:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573653.189474:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4801 00000100:00080000:0.0:1788573653.189477:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573653.189481:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573653.189491:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573653.189504:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488674176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573653.189516:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488674048:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788573653.189517:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573653.189527:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573653.189530:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573653.189535:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4801 00000100:00080000:0.0:1788573653.189539:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573653.189543:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573653.189548:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573653.189562:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488674304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573653.189571:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573653.189574:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573653.189575:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573653.189579:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4801 00000100:00080000:0.0:1788573653.189582:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573653.189584:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573653.189588:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573653.189593:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573653.189600:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488674432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573653.189608:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573653.189611:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573653.189612:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573653.189615:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4801 00000100:00080000:0.0:1788573653.189617:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573653.189619:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573653.189623:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573653.189630:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488674560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573653.189638:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573653/real 1788573653] req@ffff94d408d3c380 x1875450488674048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573653.189659:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488674048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573653.189681:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573653.189685:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573653.189689:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488674048:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573653.189714:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488674176:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573653.189738:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573653.189753:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573653/real 1788573653] req@ffff94d408d3dc00 x1875450488674176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573653.189766:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488674176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573653.189775:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573653.189778:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573653.189782:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488674176:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573653.189789:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488674304:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573653.189809:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573653.189818:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573653/real 1788573653] req@ffff94d408d3c000 x1875450488674304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573653.189827:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488674304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573653.189833:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573653.189835:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573653.189837:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488674304:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573653.189841:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488674432:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573653.189853:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573653.189859:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573653/real 1788573653] req@ffff94d408d3d180 x1875450488674432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573653.189866:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488674432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573653.189872:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573653.189874:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573653.189876:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488674432:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573653.189880:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488674560:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573653.189889:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573653.189894:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573653/real 1788573653] req@ffff94d408d3df80 x1875450488674560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573653.189901:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488674560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573653.189906:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573653.189908:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573653.189910:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488674560:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573653.701354:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573654.149358:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573654.725313:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573655.173372:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573655.749318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573656.197381:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573656.773318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573657.221348:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573657.797336:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573658.245380:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573658.309371:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573658.309387:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573658.309393:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573658.309407:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4806 00000100:00080000:0.0:1788573658.309422:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573658.309432:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573658.309493:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488674688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573658.309533:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573658.309538:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573658.309540:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573658.309545:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4806 00000100:00080000:0.0:1788573658.309549:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573658.309555:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573658.309566:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573658.309583:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488674816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573658.309592:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488674688:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788573658.309599:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573658.309605:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573658.309608:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573658.309614:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4806 00000100:00080000:0.0:1788573658.309619:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573658.309624:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573658.309630:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573658.309646:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488674944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573658.309658:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573658.309662:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573658.309665:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788573658.309668:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788573658.309674:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4806 00000100:00080000:0.0:1788573658.309681:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573658.309687:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573658.309693:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573658.309708:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488675072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573658.309714:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573658/real 1788573658] req@ffff94d408d3ce00 x1875450488674688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573658.309730:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573658.309736:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573658.309740:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573658.309740:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488674688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573658.309753:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573658.309759:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:0.0:1788573658.309760:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4806 00000100:00100000:1.0:1788573658.309764:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488674688:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788573658.309768:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573658.309776:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573658.309783:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573658.309793:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488674816:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573658.309800:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488675200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573658.309828:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573658.309839:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573658/real 1788573658] req@ffff94d408d3ea00 x1875450488674816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573658.309850:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488674816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573658.309857:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573658.309860:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573658.309863:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488674816:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573658.309867:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488674944:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573658.309881:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573658.309888:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573658/real 1788573658] req@ffff94d408d3e680 x1875450488674944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573658.309896:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488674944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573658.309902:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573658.309904:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573658.309906:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488674944:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573658.309910:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488675072:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573658.309921:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573658.309927:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573658/real 1788573658] req@ffff94d408d3d500 x1875450488675072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573658.309934:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488675072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573658.309947:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573658.309950:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573658.309952:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488675072:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573658.309963:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488675200:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573658.309975:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573658.309982:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573658/real 1788573658] req@ffff94d408d3ca80 x1875450488675200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573658.309990:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488675200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573658.309995:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573658.309997:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573658.309999:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488675200:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573658.821275:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573659.269340:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573659.845343:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573660.293349:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573660.869289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573661.317329:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573661.893349:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573662.341405:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573662.917379:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573663.365435:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573663.429365:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573663.429374:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573663.429378:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573663.429386:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4811 00000100:00080000:3.0:1788573663.429397:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573663.429404:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573663.429435:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488675328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573663.429461:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573663.429464:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573663.429465:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573663.429468:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4811 00000100:00080000:3.0:1788573663.429471:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573663.429473:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573663.429480:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573663.429488:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488675456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573663.429495:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573663.429498:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573663.429499:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573663.429502:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4811 00000100:00080000:3.0:1788573663.429505:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573663.429508:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573663.429511:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573663.429520:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488675328:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573663.429522:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488675584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573663.429535:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573663.429537:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573663.429538:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573663.429540:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4811 00000100:00080000:3.0:1788573663.429542:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573663.429545:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573663.429548:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573663.429555:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488675712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573663.429560:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573663.429562:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573663.429563:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573663.429565:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4811 00000100:00080000:3.0:1788573663.429568:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573663.429571:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573663.429574:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573663.429581:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488675840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573663.429619:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573663.429656:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573663/real 1788573663] req@ffff94d408c57100 x1875450488675328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573663.429685:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488675328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573663.429697:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573663.429702:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573663.429706:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488675328:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573663.429714:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488675456:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573663.429737:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573663.429750:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573663/real 1788573663] req@ffff94d408c54380 x1875450488675456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573663.429763:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488675456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573663.429772:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573663.429776:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573663.429780:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488675456:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573663.429803:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488675584:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573663.429824:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573663.429850:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573663/real 1788573663] req@ffff94d408c54e00 x1875450488675584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573663.429862:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488675584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573663.429870:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573663.429873:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573663.429877:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488675584:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573663.429882:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488675712:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573663.429901:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573663.429911:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573663/real 1788573663] req@ffff94d408c56d80 x1875450488675712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573663.429924:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488675712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573663.429932:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573663.429936:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573663.429940:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488675712:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573663.429947:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488675840:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573663.429968:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573663.429980:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573663/real 1788573663] req@ffff94d408c54a80 x1875450488675840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573663.429991:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488675840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573663.430006:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573663.430010:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573663.430014:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488675840:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573663.941319:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573664.389394:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573664.965375:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573665.413403:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573665.989302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573666.437385:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573667.013313:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573667.461387:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573668.037321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573668.485417:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573668.549334:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573668.549342:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573668.549346:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573668.549355:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4816 00000100:00080000:3.0:1788573668.549365:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573668.549372:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573668.549406:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488675968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573668.549432:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573668.549436:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573668.549437:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573668.549441:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4816 00000100:00080000:3.0:1788573668.549444:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573668.549449:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573668.549458:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573668.549471:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488676096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573668.549483:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488675968:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573668.549488:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573668.549500:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573668.549503:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573668.549507:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4816 00000100:00080000:3.0:1788573668.549510:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573668.549513:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573668.549517:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573668.549528:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488676224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573668.549535:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573668.549538:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573668.549539:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573668.549542:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4816 00000100:00080000:3.0:1788573668.549544:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573668.549546:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573668.549549:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573668.549555:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488676352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573668.549562:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573668.549565:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573668.549566:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573668.549568:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4816 00000100:00080000:3.0:1788573668.549570:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573668.549572:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573668.549575:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573668.549581:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488676480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573668.549600:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573668.549640:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573668/real 1788573668] req@ffff94d408c56680 x1875450488675968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573668.549658:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488675968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573668.549668:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573668.549672:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573668.549675:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488675968:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573668.549701:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488676096:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573668.549720:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573668.549729:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573668/real 1788573668] req@ffff94d408c57b80 x1875450488676096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573668.549737:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488676096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573668.549744:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573668.549746:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573668.549748:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488676096:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573668.549753:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488676224:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573668.549764:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573668.549770:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573668/real 1788573668] req@ffff94d408c54a80 x1875450488676224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573668.549777:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488676224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573668.549783:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573668.549785:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573668.549787:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488676224:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573668.549791:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488676352:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573668.549803:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573668.549809:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573668/real 1788573668] req@ffff94d408c56d80 x1875450488676352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573668.549817:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488676352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573668.549822:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573668.549824:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573668.549826:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488676352:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573668.549830:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488676480:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573668.549840:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573668.549845:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573668/real 1788573668] req@ffff94d408c54e00 x1875450488676480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573668.549852:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488676480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573668.549856:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573668.549858:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573668.549860:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488676480:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573669.061312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573669.509366:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573670.085278:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573670.533357:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573671.109339:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573671.557441:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573672.133290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573672.581321:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573673.157310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573673.605391:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573673.669394:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573673.669400:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573673.669404:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573673.669412:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4821 00000100:00080000:3.0:1788573673.669423:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573673.669430:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573673.669459:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488676608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573673.669481:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573673.669484:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573673.669485:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573673.669488:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4821 00000100:00080000:3.0:1788573673.669491:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573673.669494:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573673.669502:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573673.669512:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488676736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573673.669523:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573673.669527:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573673.669530:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573673.669532:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488676608:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573673.669537:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4821 00000100:00080000:3.0:1788573673.669543:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573673.669546:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573673.669550:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573673.669561:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488676864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573673.669568:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573673.669571:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573673.669572:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573673.669576:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4821 00000100:00080000:3.0:1788573673.669578:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573673.669581:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573673.669583:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573673.669590:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488676992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573673.669597:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573673.669599:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1788573673.669604:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573673.669604:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573673.669611:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4821 00000100:00080000:3.0:1788573673.669615:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573673.669639:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573673/real 1788573673] req@ffff94d408c54380 x1875450488676608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573673.669660:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488676608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573673.669672:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573673.669677:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573673.669686:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488676608:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573673.669686:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573673.669697:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573673.669707:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488677120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573673.669722:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488676736:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573673.669785:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573673.669811:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573673/real 1788573673] req@ffff94d408c57100 x1875450488676736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573673.669825:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488676736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573673.669838:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573673.669844:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573673.669847:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488676736:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573673.669854:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488676864:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573673.669872:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573673.669880:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573673/real 1788573673] req@ffff94d408c57800 x1875450488676864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573673.669888:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488676864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573673.669894:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573673.669898:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573673.669900:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488676864:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573673.669904:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488676992:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573673.669917:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573673.669924:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573673/real 1788573673] req@ffff94d408c55c00 x1875450488676992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573673.669935:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488676992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573673.669944:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573673.669948:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573673.669952:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488676992:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573673.669976:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488677120:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573673.669999:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573673.670010:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573673/real 1788573673] req@ffff94d408c55880 x1875450488677120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573673.670020:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488677120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573673.670026:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573673.670029:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573673.670032:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488677120:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573674.181302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573674.629355:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573675.205337:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573675.653330:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573676.229277:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573676.677328:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573677.253324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573677.701328:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573678.277284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573678.725393:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573678.789311:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573678.789322:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573678.789327:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573678.789337:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4827 00000100:00080000:3.0:1788573678.789350:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573678.789358:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573678.789402:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488677248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573678.789433:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573678.789436:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573678.789437:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573678.789440:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4827 00000100:00080000:3.0:1788573678.789443:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573678.789445:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573678.789453:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573678.789462:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488677376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573678.789476:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573678.789478:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573678.789480:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573678.789483:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4827 00000100:00080000:3.0:1788573678.789487:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573678.789491:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573678.789494:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488677248:192.168.202.125@tcp:250:kworker.0 00000008:00100000:3.0:1788573678.789494:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573678.789507:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488677504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573678.789513:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573678.789515:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573678.789516:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573678.789519:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4827 00000100:00080000:3.0:1788573678.789521:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573678.789524:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573678.789528:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573678.789554:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488677632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573678.789560:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573678.789563:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573678.789564:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573678.789568:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4827 00000100:00080000:3.0:1788573678.789572:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573678.789576:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573678.789580:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573678.789588:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488677760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573678.789599:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573678.789663:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573678/real 1788573678] req@ffff94d408c56a00 x1875450488677248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573678.789716:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488677248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573678.789726:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573678.789729:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573678.789733:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488677248:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573678.789762:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488677376:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573678.789791:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573678.789829:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573678/real 1788573678] req@ffff94d408c55180 x1875450488677376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573678.789849:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488677376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573678.789857:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573678.789860:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573678.789863:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488677376:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573678.789868:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488677504:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573678.789885:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573678.789894:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573678/real 1788573678] req@ffff94d408c57480 x1875450488677504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573678.789902:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488677504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573678.789914:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573678.789917:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573678.789920:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488677504:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573678.789925:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488677632:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573678.789937:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573678.789943:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573678/real 1788573678] req@ffff94d408c56300 x1875450488677632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573678.789950:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488677632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573678.789955:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573678.789957:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573678.789959:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488677632:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573678.789963:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488677760:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573678.789973:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573678.789979:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573678/real 1788573678] req@ffff94d408c54000 x1875450488677760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573678.789986:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488677760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573678.789991:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573678.789993:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573678.789996:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488677760:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573679.301286:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573679.749443:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573680.325277:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573680.773360:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573681.349280:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573681.797407:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573682.373289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573682.821362:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573683.397308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573683.845347:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573683.909316:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573683.909325:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573683.909329:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573683.909337:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4832 00000100:00080000:2.0:1788573683.909347:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573683.909354:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573683.909384:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488677888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573683.909407:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573683.909410:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573683.909411:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573683.909415:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4832 00000100:00080000:2.0:1788573683.909417:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573683.909419:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573683.909427:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573683.909436:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488678016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573683.909444:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573683.909449:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788573683.909450:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488677888:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788573683.909451:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573683.909459:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4832 00000100:00080000:2.0:1788573683.909461:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573683.909464:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573683.909467:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573683.909477:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488678144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573683.909490:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573683.909493:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573683.909495:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573683.909497:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4832 00000100:00080000:2.0:1788573683.909500:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573683.909503:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573683.909507:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573683.909514:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488678272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573683.909521:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573683.909523:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573683.909525:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573683.909527:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4832 00000100:00080000:2.0:1788573683.909529:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573683.909532:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573683.909535:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573683.909557:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488678400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573683.909570:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488678016:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573683.909601:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488678144:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573683.909618:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488678272:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573683.909641:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488678400:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573684.421324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573684.869395:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573685.445277:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573685.893341:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573686.469318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573686.917383:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788573686.981655:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:3.0:1788573686.981718:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788573686.981740:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:3.0:1788573686.981789:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:3.0:1788573686.981800:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:3.0:1788573686.981811:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:3.0:1788573686.981823:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:1.0:1788573686.981882:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573683/real 1788573686] req@ffff94d408c63800 x1875450488678400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573686.981931:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488678400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573686.981943:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573686.981949:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573686.981953:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488678400:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573686.981962:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573683/real 1788573686] req@ffff94d408c60000 x1875450488678016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573686.981976:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488678016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573686.981984:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573686.981988:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573686.981991:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488678016:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573686.981999:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573683/real 1788573686] req@ffff94d408c62d80 x1875450488678144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573686.982011:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488678144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573686.982020:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573686.982024:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573686.982027:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488678144:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573686.982034:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573683/real 1788573686] req@ffff94d408c62a00 x1875450488678272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573686.982046:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488678272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573686.982054:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573686.982058:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573686.982061:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488678272:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573686.982069:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573683/real 1788573686] req@ffff94d408c60a80 x1875450488677888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573686.982082:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488677888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573686.982090:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573686.982093:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573686.982096:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488677888:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573687.493314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573687.941399:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573688.517309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573688.965435:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573689.029350:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573689.029362:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573689.029366:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573689.029374:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4837 00000100:00080000:3.0:1788573689.029384:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573689.029390:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573689.029447:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488678528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573689.029485:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573689.029490:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573689.029491:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573689.029496:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4837 00000100:00080000:3.0:1788573689.029512:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573689.029518:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573689.029528:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573689.029543:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488678656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573689.029555:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573689.029558:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573689.029560:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573689.029570:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4837 00000100:00080000:3.0:1788573689.029574:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573689.029578:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573689.029583:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573689.029596:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488678784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573689.029621:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573689.029625:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573689.029627:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573689.029631:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4837 00000100:00080000:3.0:1788573689.029635:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573689.029639:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573689.029644:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573689.029656:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488678912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573689.029667:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573689.029670:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573689.029671:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573689.029675:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4837 00000100:00080000:3.0:1788573689.029678:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573689.029682:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573689.029687:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573689.029696:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488679040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573689.029799:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488678528:192.168.202.125@tcp:250:kworker.0 00000800:00000100:1.0:1788573689.029871:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573689.029916:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573689/real 1788573689] req@ffff94d408c55f80 x1875450488678528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573689.029936:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488678528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573689.029946:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573689.029951:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573689.029956:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488678528:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573689.029964:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488678656:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573689.029990:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573689.030004:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573689/real 1788573689] req@ffff94d408c54700 x1875450488678656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573689.030016:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488678656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573689.030025:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573689.030028:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573689.030032:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488678656:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573689.030037:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488678784:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573689.030057:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573689.030069:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573689/real 1788573689] req@ffff94d408c54000 x1875450488678784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573689.030081:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488678784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573689.030089:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573689.030093:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573689.030096:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488678784:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573689.030102:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488678912:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573689.030125:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573689.030136:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573689/real 1788573689] req@ffff94d408c56300 x1875450488678912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573689.030147:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488678912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573689.030155:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573689.030159:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573689.030162:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488678912:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573689.030168:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488679040:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573689.030188:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573689.030198:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573689/real 1788573689] req@ffff94d408c57480 x1875450488679040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573689.030255:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488679040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573689.030263:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573689.030266:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573689.030270:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488679040:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573689.541307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573689.989373:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573690.565373:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573691.013385:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573691.589328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573692.037432:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573692.613403:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573693.061395:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573693.637266:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573694.085326:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573694.149271:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573694.149279:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573694.149282:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573694.149314:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4842 00000100:00080000:3.0:1788573694.149323:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573694.149330:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573694.149360:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488679168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573694.149382:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573694.149385:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573694.149386:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573694.149389:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4842 00000100:00080000:3.0:1788573694.149391:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573694.149394:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573694.149401:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573694.149410:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488679296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573694.149419:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573694.149421:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488679168:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573694.149421:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573694.149425:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573694.149431:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4842 00000100:00080000:3.0:1788573694.149433:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573694.149436:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573694.149439:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573694.149449:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488679424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573694.149456:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573694.149459:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573694.149461:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573694.149464:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4842 00000100:00080000:3.0:1788573694.149467:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573694.149469:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573694.149473:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573694.149480:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488679552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573694.149485:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573694.149493:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573694.149496:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573694.149497:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573694.149501:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4842 00000100:00080000:3.0:1788573694.149505:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573694.149507:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573694.149511:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573694.149523:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573694/real 1788573694] req@ffff94d408c55180 x1875450488679168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573694.149523:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488679680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573694.149567:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488679168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573694.149579:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573694.149583:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573694.149588:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488679168:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573694.149611:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488679296:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573694.149633:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573694.149643:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573694/real 1788573694] req@ffff94d408c56a00 x1875450488679296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573694.149652:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488679296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573694.149657:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573694.149660:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573694.149662:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488679296:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573694.149666:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488679424:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573694.149676:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573694.149682:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573694/real 1788573694] req@ffff94d408c55880 x1875450488679424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573694.149689:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488679424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573694.149695:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573694.149696:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573694.149698:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488679424:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573694.149702:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488679552:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573694.149711:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573694.149717:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573694/real 1788573694] req@ffff94d408c55c00 x1875450488679552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573694.149725:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488679552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573694.149730:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573694.149731:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573694.149733:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488679552:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573694.149736:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488679680:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573694.149745:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573694.149750:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573694/real 1788573694] req@ffff94d408c57800 x1875450488679680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573694.149757:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488679680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573694.149762:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573694.149764:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573694.149766:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488679680:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573694.661351:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573695.109422:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573695.685635:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573696.133344:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573696.709356:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573697.157418:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573697.733312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573698.181359:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573698.757335:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573699.205430:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573699.269329:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573699.269338:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573699.269343:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573699.269352:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4847 00000100:00080000:1.0:1788573699.269362:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573699.269368:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573699.269408:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488679808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573699.269440:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573699.269444:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573699.269447:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573699.269452:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4847 00000100:00080000:1.0:1788573699.269457:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573699.269462:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573699.269473:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573699.269473:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488679808:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573699.269494:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488679936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573699.269508:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573699.269512:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573699.269514:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573699.269518:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4847 00000100:00080000:1.0:1788573699.269524:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573699.269528:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573699.269534:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788573699.269538:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573699.269551:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488680064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573699.269577:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573699/real 1788573699] req@ffff94d4049bd500 x1875450488679808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573699.269587:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573699.269592:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573699.269594:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573699.269599:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488679808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573699.269603:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4847 00000100:00080000:1.0:1788573699.269611:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573699.269615:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573699.269619:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00080000:3.0:1788573699.269622:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573699.269627:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488679808:192.168.202.125@tcp:250:kworker.0 00000008:00100000:1.0:1788573699.269628:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573699.269646:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488680192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573699.269649:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488679936:192.168.202.125@tcp:38:kworker.0 00000100:00080000:1.0:1788573699.269667:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573699.269670:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573699.269672:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573699.269676:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4847 00000100:00080000:1.0:1788573699.269680:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788573699.269685:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788573699.269694:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573699.269699:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573699.269705:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573699/real 1788573699] req@ffff94d4049bf100 x1875450488679936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573699.269712:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488680320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573699.269724:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488679936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573699.269736:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573699.269741:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573699.269745:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488679936:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573699.269753:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488680064:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573699.269774:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573699.269783:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573699/real 1788573699] req@ffff94d4049bca80 x1875450488680064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573699.269791:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488680064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573699.269803:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573699.269807:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573699.269811:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488680064:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573699.269836:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488680192:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573699.269861:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573699.269870:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573699/real 1788573699] req@ffff94d4049be680 x1875450488680192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573699.269878:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488680192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573699.269884:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573699.269887:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573699.269889:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488680192:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573699.269893:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488680320:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573699.269916:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573699.269928:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573699/real 1788573699] req@ffff94d4049bdc00 x1875450488680320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573699.269942:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488680320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573699.269951:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573699.269954:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573699.269957:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488680320:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573699.781330:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573700.229342:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573700.805281:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573701.253368:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573701.829280:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573702.277329:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573702.853272:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573703.301388:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573703.877300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573704.325411:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573704.389337:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573704.389353:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573704.389359:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573704.389374:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4852 00000100:00080000:1.0:1788573704.389388:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573704.389398:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573704.389451:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488680448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573704.389489:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573704.389494:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573704.389496:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573704.389502:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4852 00000100:00080000:1.0:1788573704.389506:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573704.389512:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573704.389524:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573704.389540:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488680448:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573704.389544:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd880 x1875450488680576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573704.389559:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573704.389563:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573704.389566:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573704.389570:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4852 00000100:00080000:1.0:1788573704.389575:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573704.389581:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573704.389588:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573704.389605:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488680704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573704.389616:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573704.389622:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1788573704.389623:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788573704.389625:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573704.389633:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4852 00000100:00080000:1.0:1788573704.389638:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573704.389644:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573704.389650:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573704.389665:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488680832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573704.389675:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573704/real 1788573704] req@ffff94d4049be300 x1875450488680448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573704.389679:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573704.389690:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573704.389693:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573704.389699:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4852 00000100:00100000:3.0:1788573704.389703:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488680448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573704.389704:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573704.389713:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00080000:3.0:1788573704.389718:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1788573704.389722:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788573704.389723:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573704.389729:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488680448:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573704.389740:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488680960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573704.389757:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488680576:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573704.389778:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573704.389789:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573704/real 1788573704] req@ffff94d4049bd880 x1875450488680576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573704.389800:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd880 x1875450488680576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573704.389806:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573704.389809:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573704.389812:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488680576:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573704.389816:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488680704:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573704.389827:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573704.389833:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573704/real 1788573704] req@ffff94d4049bdc00 x1875450488680704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573704.389840:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488680704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573704.389846:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573704.389848:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573704.389849:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488680704:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573704.389853:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488680832:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573704.389865:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573704.389870:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573704/real 1788573704] req@ffff94d4049be680 x1875450488680832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573704.389878:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488680832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573704.389883:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573704.389885:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573704.389887:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488680832:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573704.389890:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488680960:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573704.389899:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573704.389905:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573704/real 1788573704] req@ffff94d4049bca80 x1875450488680960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573704.389912:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488680960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573704.389918:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573704.389919:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573704.389921:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488680960:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573704.901259:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573705.349347:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573705.925289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573706.373427:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573706.949269:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573707.397333:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573707.973339:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573708.421441:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573708.997367:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573709.445454:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573709.509317:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573709.509329:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573709.509334:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573709.509346:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4857 00000100:00080000:1.0:1788573709.509359:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573709.509367:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573709.509420:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488681088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573709.509464:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573709.509469:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573709.509471:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573709.509477:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4857 00000100:00080000:1.0:1788573709.509482:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573709.509487:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573709.509499:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573709.509517:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488681216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573709.509533:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788573709.509533:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488681088:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788573709.509551:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573709.509555:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573709.509563:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4857 00000100:00080000:1.0:1788573709.509568:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573709.509574:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573709.509581:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573709.509601:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488681344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573709.509614:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788573709.509621:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788573709.509622:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573709.509626:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573709.509634:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4857 00000100:00080000:1.0:1788573709.509641:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573709.509648:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573709.509657:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573709.509663:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573709/real 1788573709] req@ffff94d4049bf100 x1875450488681088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573709.509682:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488681088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573709.509697:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488681472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573709.509708:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573709.509715:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573709.509724:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488681088:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788573709.509726:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573709.509732:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573709.509737:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573709.509743:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4857 00000100:00080000:1.0:1788573709.509749:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788573709.509750:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488681216:192.168.202.125@tcp:38:kworker.0 00000100:00080000:1.0:1788573709.509761:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573709.509770:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788573709.509775:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573709.509793:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573709/real 1788573709] req@ffff94d4049bd500 x1875450488681216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573709.509800:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488681600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573709.509830:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488681216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573709.509843:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573709.509847:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573709.509850:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488681216:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573709.509857:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488681344:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573709.509891:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573709.509928:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573709/real 1788573709] req@ffff94d4049bf800 x1875450488681344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573709.509941:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488681344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573709.509954:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573709.509961:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573709.509965:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488681344:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573709.509974:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488681472:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573709.510003:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573709.510013:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573709/real 1788573709] req@ffff94d4049bed80 x1875450488681472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573709.510022:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488681472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573709.510029:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573709.510031:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573709.510034:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488681472:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573709.510064:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488681600:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573709.510100:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573709.510120:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573709/real 1788573709] req@ffff94d4049bc700 x1875450488681600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573709.510140:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488681600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573709.510148:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573709.510152:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573709.510154:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488681600:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573710.021298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573710.469416:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573711.045344:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573711.493421:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573712.069342:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573712.517404:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573713.093290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573713.541400:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573714.117321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573714.565468:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573714.629330:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573714.629339:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573714.629342:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573714.629352:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4862 00000100:00080000:3.0:1788573714.629361:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573714.629367:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573714.629400:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488681728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573714.629428:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573714.629431:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573714.629432:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573714.629435:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4862 00000100:00080000:3.0:1788573714.629438:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573714.629442:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573714.629453:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573714.629467:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488681856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573714.629484:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573714.629487:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488681728:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573714.629490:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573714.629495:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573714.629510:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4862 00000100:00080000:3.0:1788573714.629514:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573714.629518:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573714.629522:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573714.629536:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488681984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573714.629545:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573714.629549:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573714.629551:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573714.629557:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4862 00000100:00080000:3.0:1788573714.629562:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573714.629567:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573714.629574:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573714.629587:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573714.629598:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488682112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573714.629609:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573714.629612:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573714.629614:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573714.629618:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4862 00000100:00080000:3.0:1788573714.629623:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573714.629628:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573714.629632:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573714.629635:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573714/real 1788573714] req@ffff94d408c57100 x1875450488681728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573714.629646:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488682240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573714.629669:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488681728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573714.629683:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573714.629687:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573714.629691:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488681728:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573714.629725:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488681856:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573714.629753:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573714.629765:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573714/real 1788573714] req@ffff94d408c54380 x1875450488681856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573714.629788:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488681856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573714.629795:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573714.629797:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573714.629800:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488681856:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573714.629805:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488681984:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573714.629830:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573714.629843:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573714/real 1788573714] req@ffff94d408c54e00 x1875450488681984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573714.629857:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488681984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573714.629866:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573714.629870:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573714.629873:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488681984:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573714.629878:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488682112:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573714.629896:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573714.629914:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573714/real 1788573714] req@ffff94d408c56d80 x1875450488682112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573714.629927:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488682112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573714.629936:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573714.629940:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573714.629943:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488682112:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573714.629951:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488682240:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573714.629977:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573714.629985:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573714/real 1788573714] req@ffff94d408c54a80 x1875450488682240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573714.629999:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488682240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573714.630004:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573714.630006:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573714.630009:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488682240:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573715.141289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573715.589345:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573716.165324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573716.613435:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573717.189298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573717.637391:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573718.213338:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573718.661386:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573719.237277:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573719.685371:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573719.749387:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573719.749399:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573719.749403:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573719.749414:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4868 00000100:00080000:0.0:1788573719.749427:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573719.749435:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573719.749483:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f800 x1875450488682368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573719.749511:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573719.749517:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573719.749519:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573719.749522:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4868 00000100:00080000:0.0:1788573719.749524:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573719.749527:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573719.749535:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573719.749544:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f100 x1875450488682496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573719.749553:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573719.749556:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573719.749557:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573719.749563:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4868 00000100:00080000:0.0:1788573719.749565:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573719.749568:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573719.749571:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573719.749578:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488682624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573719.749586:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573719.749589:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573719.749591:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573719.749595:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4868 00000100:00080000:0.0:1788573719.749601:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573719.749604:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488682368:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788573719.749605:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573719.749615:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573719.749627:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488682752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573719.749636:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573719.749639:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573719.749640:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573719.749643:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4868 00000100:00080000:0.0:1788573719.749645:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573719.749648:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573719.749650:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573719.749658:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488682880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573719.749697:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573719.749734:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573719/real 1788573719] req@ffff94d408d3f800 x1875450488682368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573719.749755:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f800 x1875450488682368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573719.749769:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573719.749773:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573719.749778:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488682368:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573719.749788:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488682496:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573719.749816:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573719.749831:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573719/real 1788573719] req@ffff94d408d3f100 x1875450488682496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573719.749846:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f100 x1875450488682496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573719.749855:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573719.749859:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573719.749863:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488682496:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573719.749885:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488682624:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573719.749906:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573719.749919:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573719/real 1788573719] req@ffff94d408d3ca80 x1875450488682624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573719.749933:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488682624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573719.749942:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573719.749946:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573719.749949:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488682624:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573719.749957:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488682752:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573719.749978:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573719.749987:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573719/real 1788573719] req@ffff94d408d3d500 x1875450488682752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573719.749995:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488682752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573719.750005:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573719.750007:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573719.750009:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488682752:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573719.750013:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488682880:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573719.750025:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573719.750030:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573719/real 1788573719] req@ffff94d408d3e680 x1875450488682880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573719.750037:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488682880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573719.750042:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573719.750044:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573719.750046:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488682880:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573720.261400:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573720.709384:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573721.285314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573721.733423:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573722.309333:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573722.757378:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573723.333303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573723.781387:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573724.357299:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573724.805365:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573724.869332:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573724.869343:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573724.869347:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573724.869359:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4873 00000100:00080000:3.0:1788573724.869372:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573724.869380:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573724.869427:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488683008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573724.869455:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573724.869459:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573724.869460:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573724.869463:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4873 00000100:00080000:3.0:1788573724.869465:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573724.869468:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573724.869477:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573724.869486:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488683136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573724.869495:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573724.869498:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573724.869499:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573724.869502:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4873 00000100:00080000:3.0:1788573724.869504:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573724.869507:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573724.869510:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573724.869518:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488683008:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573724.869522:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488683264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573724.869535:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573724.869537:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573724.869538:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573724.869541:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4873 00000100:00080000:3.0:1788573724.869542:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573724.869545:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573724.869548:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573724.869557:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488683392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573724.869580:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573724.869583:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573724.869585:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573724.869603:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4873 00000100:00080000:3.0:1788573724.869606:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573724.869609:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573724.869613:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573724.869626:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488683520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573724.869658:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573724.869706:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573724/real 1788573724] req@ffff94d408c57b80 x1875450488683008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573724.869730:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488683008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573724.869755:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573724.869759:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573724.869765:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488683008:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573724.869773:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488683136:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573724.869798:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573724.869811:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573724/real 1788573724] req@ffff94d408c56680 x1875450488683136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573724.869826:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488683136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573724.869836:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573724.869840:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573724.869843:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488683136:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573724.869884:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488683264:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573724.869909:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573724.869922:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573724/real 1788573724] req@ffff94d408c54a80 x1875450488683264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573724.869936:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488683264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573724.869945:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573724.869949:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573724.869951:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488683264:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573724.869957:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488683392:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573724.869974:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573724.869982:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573724/real 1788573724] req@ffff94d408c56d80 x1875450488683392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573724.869990:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488683392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573724.869996:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573724.869998:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573724.870000:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488683392:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573724.870004:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488683520:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573724.870014:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573724.870019:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573724/real 1788573724] req@ffff94d408c54e00 x1875450488683520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573724.870026:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488683520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573724.870031:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573724.870033:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573724.870035:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488683520:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573725.381327:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573725.829428:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573726.405281:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573726.853432:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573727.429335:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573727.877567:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573728.453272:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573728.901401:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573729.477310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573729.925425:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573729.989276:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573729.989292:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573729.989296:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573729.989308:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4878 00000100:00080000:3.0:1788573729.989320:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573729.989328:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573729.989371:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488683648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573729.989399:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573729.989403:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573729.989404:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573729.989407:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4878 00000100:00080000:3.0:1788573729.989410:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573729.989419:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573729.989430:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573729.989445:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488683776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573729.989476:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573729.989480:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488683648:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573729.989482:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573729.989487:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573729.989492:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4878 00000100:00080000:3.0:1788573729.989495:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573729.989498:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573729.989503:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573729.989524:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488683904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573729.989532:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573729.989534:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573729.989536:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573729.989538:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4878 00000100:00080000:3.0:1788573729.989549:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573729.989552:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573729.989556:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573729.989559:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573729.989569:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488684032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573729.989577:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573729.989580:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573729.989582:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573729.989584:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4878 00000100:00080000:3.0:1788573729.989587:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573729.989590:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573729.989599:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573729.989603:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573729/real 1788573729] req@ffff94d408c54380 x1875450488683648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573729.989610:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488684160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573729.989633:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488683648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573729.989643:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573729.989646:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573729.989657:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488683648:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573729.989687:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488683776:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573729.989713:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573729.989724:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573729/real 1788573729] req@ffff94d408c57100 x1875450488683776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573729.989733:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488683776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573729.989739:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573729.989742:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573729.989744:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488683776:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573729.989748:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488683904:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573729.989761:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573729.989768:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573729/real 1788573729] req@ffff94d408c57800 x1875450488683904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573729.989776:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488683904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573729.989782:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573729.989784:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573729.989786:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488683904:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573729.989789:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488684032:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573729.989801:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573729.989806:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573729/real 1788573729] req@ffff94d408c55c00 x1875450488684032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573729.989815:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488684032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573729.989823:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573729.989827:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573729.989830:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488684032:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573729.989836:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488684160:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573729.989857:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573729.989868:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573729/real 1788573729] req@ffff94d408c55880 x1875450488684160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573729.989877:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488684160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573729.989888:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573729.989890:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573729.989892:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488684160:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573730.501332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573730.949413:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573731.525324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573731.973339:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573732.549285:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573732.997396:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573733.573312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573734.021417:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573734.597332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573735.045381:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573735.109301:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573735.109312:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573735.109316:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573735.109326:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4883 00000100:00080000:3.0:1788573735.109338:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573735.109345:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573735.109395:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488684288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573735.109424:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573735.109429:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573735.109431:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573735.109435:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4883 00000100:00080000:3.0:1788573735.109438:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573735.109443:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573735.109453:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573735.109467:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488684416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573735.109479:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573735.109482:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573735.109485:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573735.109491:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4883 00000100:00100000:1.0:1788573735.109497:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488684288:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573735.109498:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573735.109509:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573735.109525:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573735.109547:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488684544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573735.109558:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573735.109561:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573735.109564:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573735.109568:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4883 00000100:00080000:3.0:1788573735.109574:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573735.109578:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573735.109584:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573735.109598:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488684672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573735.109614:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573735.109620:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573735.109626:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573735.109628:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573735.109634:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4883 00000100:00080000:3.0:1788573735.109641:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573735.109646:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573735.109650:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573735.109664:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488684800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573735.109678:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573735/real 1788573735] req@ffff94d408c56a00 x1875450488684288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573735.109708:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488684288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573735.109721:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573735.109726:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573735.109731:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488684288:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573735.109764:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488684416:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573735.109792:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573735.109806:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573735/real 1788573735] req@ffff94d408c55180 x1875450488684416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573735.109819:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488684416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573735.109839:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573735.109843:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573735.109851:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488684416:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573735.109858:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488684544:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788573735.109878:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573735.109891:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573735/real 1788573735] req@ffff94d408c57480 x1875450488684544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573735.109904:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488684544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573735.109914:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573735.109917:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573735.109920:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488684544:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788573735.109928:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488684672:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573735.109948:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573735.109959:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573735/real 1788573735] req@ffff94d408c56300 x1875450488684672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573735.109972:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488684672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573735.109980:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573735.109984:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573735.109988:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488684672:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573735.109994:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488684800:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788573735.110013:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573735.110024:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573735/real 1788573735] req@ffff94d408c54000 x1875450488684800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573735.110037:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488684800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573735.110046:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573735.110049:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788573735.110053:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488684800:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573735.621306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573736.069373:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573736.645258:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573737.093387:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573737.669372:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573738.117378:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573738.693346:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573739.141364:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573739.717336:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573740.165323:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573740.229363:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573740.229372:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573740.229376:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573740.229385:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4888 00000100:00080000:1.0:1788573740.229395:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573740.229402:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573740.229442:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488684928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573740.229469:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573740.229472:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573740.229473:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573740.229476:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4888 00000100:00080000:1.0:1788573740.229479:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573740.229483:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573740.229491:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573740.229502:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488685056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573740.229509:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488684928:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788573740.229512:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573740.229517:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573740.229520:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573740.229524:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4888 00000100:00080000:1.0:1788573740.229526:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573740.229528:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573740.229531:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573740.229541:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488685184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573740.229547:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573740.229549:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573740.229551:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573740.229553:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4888 00000100:00080000:1.0:1788573740.229563:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573740.229566:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573740.229571:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788573740.229583:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573740.229584:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488685312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573740.229592:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573740.229595:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573740.229596:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573740.229598:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4888 00000100:00080000:1.0:1788573740.229601:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573740.229604:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573740.229608:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573740.229618:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488685440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573740.229626:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573740/real 1788573740] req@ffff94d4049bc000 x1875450488684928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573740.229648:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488684928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573740.229660:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573740.229664:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573740.229668:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488684928:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573740.229693:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488685056:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573740.229714:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573740.229726:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573740/real 1788573740] req@ffff94d4049bfb80 x1875450488685056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573740.229737:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488685056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573740.229745:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573740.229748:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573740.229751:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488685056:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573740.229756:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488685184:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573740.229775:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573740.229784:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573740/real 1788573740] req@ffff94d4049bea00 x1875450488685184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573740.229796:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488685184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573740.229804:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573740.229808:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573740.229811:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488685184:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573740.229817:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488685312:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573740.229837:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573740.229848:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573740/real 1788573740] req@ffff94d4049bf480 x1875450488685312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573740.229860:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488685312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573740.229867:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573740.229871:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573740.229874:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488685312:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573740.229880:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488685440:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573740.229899:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573740.229910:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573740/real 1788573740] req@ffff94d4049bc380 x1875450488685440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573740.229922:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc380 x1875450488685440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573740.229928:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573740.229929:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573740.229931:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488685440:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573740.741295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573741.189359:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573741.765342:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573742.213378:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573742.789354:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573743.237410:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573743.813293:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573744.261362:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573744.837270:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573745.285342:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573745.349347:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573745.349360:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573745.349365:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573745.349377:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4893 00000100:00080000:2.0:1788573745.349394:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573745.349403:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573745.349442:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488685568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573745.349470:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573745.349473:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573745.349475:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573745.349478:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4893 00000100:00080000:2.0:1788573745.349480:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573745.349483:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573745.349492:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573745.349502:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488685696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573745.349509:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573745.349512:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573745.349514:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573745.349517:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4893 00000100:00080000:2.0:1788573745.349519:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573745.349522:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573745.349525:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573745.349533:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488685824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573745.349539:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573745.349541:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573745.349543:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573745.349546:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4893 00000100:00080000:2.0:1788573745.349551:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573745.349554:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573745.349558:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573745.349566:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60e00 x1875450488685952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573745.349572:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573745.349574:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573745.349576:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573745.349580:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4893 00000100:00080000:2.0:1788573745.349584:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573745.349586:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573745.349590:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573745.349598:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63100 x1875450488686080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573745.349621:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488685568:192.168.202.125@tcp:250:kworker.0 00000800:00000100:3.0:1788573745.349772:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573745.349813:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573745/real 1788573745] req@ffff94d408c61c00 x1875450488685568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573745.349951:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488685568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573745.349967:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573745.349972:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573745.349978:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488685568:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573745.350026:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488685696:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573745.350052:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573745.350066:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573745/real 1788573745] req@ffff94d408c63b80 x1875450488685696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573745.350080:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488685696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573745.350090:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573745.350093:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573745.350097:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488685696:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573745.350104:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488685824:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573745.350121:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573745.350131:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573745/real 1788573745] req@ffff94d408c61500 x1875450488685824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573745.350143:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488685824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573745.350152:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573745.350156:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573745.350176:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488685824:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573745.350182:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488685952:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573745.350231:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573745.350246:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573745/real 1788573745] req@ffff94d408c60e00 x1875450488685952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573745.350260:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60e00 x1875450488685952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573745.350277:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573745.350281:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573745.350285:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488685952:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573745.350316:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488686080:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573745.350333:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573745.350343:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573745/real 1788573745] req@ffff94d408c63100 x1875450488686080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573745.350355:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63100 x1875450488686080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573745.350364:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573745.350367:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573745.350371:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488686080:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573745.861299:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573746.309377:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573746.885351:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573747.333407:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573747.909337:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573748.357419:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573748.933365:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573749.381435:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573749.957329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573750.405407:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573750.469303:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573750.469311:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573750.469315:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573750.469324:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4898 00000100:00080000:2.0:1788573750.469332:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573750.469339:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573750.469375:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488686208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573750.469405:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573750.469412:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573750.469414:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573750.469419:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4898 00000100:00080000:2.0:1788573750.469424:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573750.469428:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573750.469436:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573750.469448:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488686208:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788573750.469450:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488686336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573750.469466:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573750.469470:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573750.469472:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573750.469477:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4898 00000100:00080000:2.0:1788573750.469482:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573750.469486:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573750.469492:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573750.469507:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488686464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573750.469515:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573750.469518:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573750.469520:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573750.469524:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4898 00000100:00080000:2.0:1788573750.469529:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573750.469532:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573750.469538:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573750.469539:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488686336:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788573750.469553:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63100 x1875450488686592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573750.469574:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488686464:192.168.202.125@tcp:38:kworker.0 00000100:00080000:2.0:1788573750.469576:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573750.469585:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573750.469588:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573750.469594:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4898 00000100:00100000:3.0:1788573750.469598:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488686592:192.168.202.125@tcp:8:kworker.0 00000100:00080000:2.0:1788573750.469601:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573750.469607:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573750.469612:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573750.469625:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60e00 x1875450488686720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573750.469661:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488686720:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573750.981345:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573751.429399:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573752.005288:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573752.453377:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573753.029320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573753.477363:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788573753.541498:0:2392:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:1.0:1788573753.541539:0:2392:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788573753.541555:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788573753.541597:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788573753.541606:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788573753.541614:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788573753.541622:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:3.0:1788573753.541695:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573750/real 1788573753] req@ffff94d408c60e00 x1875450488686720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573753.541745:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60e00 x1875450488686720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573753.541760:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573753.541767:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573753.541772:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488686720:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573753.541783:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573750/real 1788573753] req@ffff94d408c63100 x1875450488686592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573753.541797:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63100 x1875450488686592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573753.541806:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573753.541810:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573753.541814:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488686592:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573753.541823:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573750/real 1788573753] req@ffff94d408c61880 x1875450488686336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573753.541836:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488686336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573753.541846:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573753.541851:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573753.541854:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488686336:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573753.541863:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573750/real 1788573753] req@ffff94d408c61f80 x1875450488686464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573753.541878:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488686464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573753.541888:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573753.541892:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573753.541896:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488686464:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573753.541904:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573750/real 1788573753] req@ffff94d408c63480 x1875450488686208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573753.541919:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488686208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573753.541928:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573753.541932:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573753.541936:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488686208:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573754.053300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573754.501388:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573755.077342:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573755.525429:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573755.589408:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573755.589424:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573755.589430:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573755.589442:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4903 00000100:00080000:1.0:1788573755.589457:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573755.589467:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573755.589513:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488686848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573755.589544:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573755.589549:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573755.589551:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573755.589555:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4903 00000100:00080000:1.0:1788573755.589559:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573755.589563:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573755.589576:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573755.589598:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488686976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573755.589606:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488686848:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788573755.589620:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573755.589625:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573755.589627:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573755.589633:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4903 00000100:00080000:1.0:1788573755.589637:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573755.589642:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573755.589649:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573755.589666:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488687104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573755.589684:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573755.589688:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573755.589690:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573755.589698:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4903 00000800:00000100:3.0:1788573755.589698:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788573755.589706:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573755.589713:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573755.589721:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573755.589736:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488687232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573755.589748:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573755.589752:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573755.589758:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573755.589759:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573755/real 1788573755] req@ffff94d4049bd180 x1875450488686848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573755.589768:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4903 00000100:00080000:1.0:1788573755.589778:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573755.589786:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573755.589793:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573755.589796:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488686848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573755.589810:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488687360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573755.589813:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573755.589822:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573755.589831:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488686848:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573755.589873:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488686976:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573755.589903:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573755.589918:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573755/real 1788573755] req@ffff94d4049bce00 x1875450488686976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573755.589932:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488686976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573755.589942:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573755.589945:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573755.589948:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488686976:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573755.589971:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488687104:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573755.589995:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573755.590008:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573755/real 1788573755] req@ffff94d4049bc380 x1875450488687104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573755.590021:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc380 x1875450488687104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573755.590029:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573755.590033:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573755.590037:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488687104:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573755.590043:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488687232:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573755.590066:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573755.590078:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573755/real 1788573755] req@ffff94d4049bf480 x1875450488687232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573755.590090:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488687232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573755.590099:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573755.590102:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573755.590105:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488687232:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573755.590112:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488687360:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573755.590134:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573755.590146:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573755/real 1788573755] req@ffff94d4049bea00 x1875450488687360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573755.590161:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488687360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573755.590324:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573755.590329:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573755.590334:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488687360:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573756.101312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573756.549370:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573757.125361:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573757.573385:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573758.149274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573758.597344:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573759.173322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573759.621376:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573760.197312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573760.645322:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573760.709374:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573760.709390:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573760.709396:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573760.709411:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4909 00000100:00080000:2.0:1788573760.709426:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573760.709436:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573760.709492:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488687488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573760.709531:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573760.709536:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573760.709538:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573760.709543:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4909 00000100:00080000:2.0:1788573760.709548:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573760.709554:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573760.709565:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573760.709582:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488687616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573760.709590:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488687488:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788573760.709600:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573760.709605:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573760.709607:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573760.709613:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4909 00000100:00080000:2.0:1788573760.709617:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573760.709621:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573760.709627:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573760.709641:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488687744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573760.709652:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573760.709657:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573760.709658:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573760.709662:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4909 00000100:00080000:2.0:1788573760.709669:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573760.709674:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573760.709680:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788573760.709686:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573760.709697:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488687872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573760.709716:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573760.709720:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573760.709722:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573760.709726:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4909 00000100:00080000:2.0:1788573760.709731:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573760.709736:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573760.709741:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573760.709749:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573760/real 1788573760] req@ffff94d408c61500 x1875450488687488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573760.709754:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488688000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573760.709792:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488687488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573760.709814:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573760.709819:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573760.709824:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488687488:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573760.709868:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488687616:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573760.709897:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573760.709909:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573760/real 1788573760] req@ffff94d408c63b80 x1875450488687616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573760.709919:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488687616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573760.709925:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573760.709928:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573760.709930:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488687616:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573760.709935:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488687744:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573760.709951:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573760.709958:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573760/real 1788573760] req@ffff94d408c61c00 x1875450488687744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573760.709965:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488687744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573760.709971:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573760.709974:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573760.709976:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488687744:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573760.709980:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488687872:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573760.709997:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573760.710008:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573760/real 1788573760] req@ffff94d408c60a80 x1875450488687872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573760.710021:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488687872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573760.710031:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573760.710035:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573760.710038:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488687872:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573760.710045:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488688000:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573760.710064:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573760.710072:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573760/real 1788573760] req@ffff94d408c62a00 x1875450488688000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573760.710080:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488688000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573760.710086:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573760.710088:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573760.710090:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488688000:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573761.221337:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573761.669388:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573762.245310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573762.693469:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573763.269327:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573763.717353:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573764.293321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573764.741427:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573765.317314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573765.765372:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573765.829304:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573765.829312:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573765.829316:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573765.829324:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4914 00000100:00080000:2.0:1788573765.829334:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573765.829341:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573765.829381:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488688128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573765.829406:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573765.829408:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573765.829410:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573765.829413:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4914 00000100:00080000:2.0:1788573765.829416:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573765.829418:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573765.829427:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573765.829436:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488688256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573765.829445:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573765.829447:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573765.829449:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573765.829454:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488688128:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788573765.829455:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4914 00000100:00080000:2.0:1788573765.829460:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573765.829464:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573765.829467:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573765.829481:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488688384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573765.829488:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573765.829491:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573765.829492:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573765.829496:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4914 00000100:00080000:2.0:1788573765.829505:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573765.829508:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573765.829511:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788573765.829515:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573765.829524:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488688512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573765.829535:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573765.829539:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573765.829541:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573765.829546:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4914 00000100:00100000:3.0:1788573765.829549:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573765/real 1788573765] req@ffff94d408c62d80 x1875450488688128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573765.829552:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573765.829562:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573765.829568:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573765.829570:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488688128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573765.829581:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1788573765.829586:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488688640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573765.829586:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573765.829592:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488688128:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573765.829612:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488688256:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573765.829628:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573765.829637:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573765/real 1788573765] req@ffff94d408c60000 x1875450488688256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573765.829645:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488688256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573765.829650:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573765.829652:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573765.829654:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488688256:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573765.829658:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488688384:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573765.829667:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573765.829673:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573765/real 1788573765] req@ffff94d408c63800 x1875450488688384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573765.829680:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488688384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573765.829690:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573765.829692:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573765.829694:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488688384:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573765.829697:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488688512:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573765.829707:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573765.829712:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573765/real 1788573765] req@ffff94d408c62680 x1875450488688512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573765.829719:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488688512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573765.829724:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573765.829726:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573765.829728:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488688512:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573765.829731:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488688640:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573765.829742:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573765.829748:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573765/real 1788573765] req@ffff94d408c61180 x1875450488688640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573765.829771:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488688640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573765.829777:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573765.829779:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573765.829782:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488688640:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573766.341343:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573766.789414:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573767.365287:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573767.813435:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573768.389301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573768.837428:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573769.413283:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573769.861391:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573770.437264:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573770.885396:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573770.949392:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573770.949403:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573770.949408:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573770.949420:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4919 00000100:00080000:2.0:1788573770.949433:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573770.949441:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573770.949488:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62300 x1875450488688768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573770.949519:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573770.949522:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573770.949524:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573770.949527:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4919 00000100:00080000:2.0:1788573770.949529:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573770.949532:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573770.949541:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573770.949550:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60380 x1875450488688896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573770.949562:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573770.949566:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573770.949567:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573770.949570:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4919 00000100:00080000:2.0:1788573770.949573:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573770.949575:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573770.949578:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573770.949584:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60700 x1875450488689024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573770.949593:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573770.949597:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573770.949599:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573770.949604:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4919 00000100:00080000:2.0:1788573770.949608:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573770.949613:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573770.949620:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573770.949626:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488688768:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788573770.949639:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488689152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573770.949668:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573770.949671:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573770.949672:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573770.949675:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4919 00000100:00080000:2.0:1788573770.949679:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573770.949684:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573770.949689:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573770.949705:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488689280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573770.949731:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573770.949773:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573770/real 1788573770] req@ffff94d408c62300 x1875450488688768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573770.949789:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62300 x1875450488688768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573770.949798:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573770.949801:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573770.949805:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488688768:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573770.949812:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488688896:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573770.949833:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573770.949841:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573770/real 1788573770] req@ffff94d408c60380 x1875450488688896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573770.949850:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60380 x1875450488688896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573770.949856:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573770.949858:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573770.949861:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488688896:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573770.949865:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488689024:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573770.949876:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573770.949882:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573770/real 1788573770] req@ffff94d408c60700 x1875450488689024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573770.949890:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60700 x1875450488689024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573770.949895:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573770.949897:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573770.949899:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488689024:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573770.949921:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488689152:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573770.949932:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573770.949938:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573770/real 1788573770] req@ffff94d408c61180 x1875450488689152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573770.949945:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488689152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573770.949950:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573770.949952:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573770.949954:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488689152:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573770.949958:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488689280:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573770.949967:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573770.949973:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573770/real 1788573770] req@ffff94d408c62680 x1875450488689280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573770.949985:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488689280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573770.949990:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573770.949992:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573770.949994:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488689280:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573771.461291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573771.909401:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573772.485298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573772.933395:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573773.509326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573773.957382:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573774.533370:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573774.981441:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573775.557296:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573776.005378:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573776.069364:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573776.069375:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573776.069379:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573776.069390:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4924 00000100:00080000:0.0:1788573776.069403:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573776.069412:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573776.069477:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488689408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573776.069510:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573776.069513:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573776.069519:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573776.069522:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4924 00000100:00080000:0.0:1788573776.069526:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573776.069530:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573776.069540:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573776.069566:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488689536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573776.069575:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488689408:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788573776.069578:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573776.069585:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573776.069586:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573776.069590:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4924 00000100:00080000:0.0:1788573776.069593:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573776.069596:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573776.069600:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573776.069613:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488689664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573776.069625:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573776.069628:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573776.069629:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573776.069632:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4924 00000100:00080000:0.0:1788573776.069635:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573776.069638:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573776.069641:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573776.069649:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488689792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573776.069655:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788573776.069662:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573776.069665:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573776.069667:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573776.069670:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4924 00000100:00080000:0.0:1788573776.069673:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573776.069676:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573776.069680:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573776.069686:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488689920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573776.069703:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573776/real 1788573776] req@ffff94d408d3ea00 x1875450488689408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573776.069721:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488689408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573776.069731:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573776.069734:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573776.069737:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488689408:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573776.069760:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488689536:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573776.069776:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573776.069784:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573776/real 1788573776] req@ffff94d408d3ce00 x1875450488689536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573776.069793:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488689536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573776.069804:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573776.069806:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573776.069809:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488689536:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573776.069812:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488689664:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573776.069823:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573776.069829:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573776/real 1788573776] req@ffff94d408d3df80 x1875450488689664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573776.069836:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488689664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573776.069842:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573776.069844:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573776.069846:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488689664:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573776.069850:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488689792:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573776.069862:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573776.069872:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573776/real 1788573776] req@ffff94d408d3d180 x1875450488689792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573776.069885:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488689792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573776.069893:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573776.069907:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573776.069912:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488689792:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573776.069919:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488689920:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573776.069941:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573776.069950:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573776/real 1788573776] req@ffff94d408d3c000 x1875450488689920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573776.069959:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488689920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573776.069964:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573776.069966:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573776.069969:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488689920:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573776.581327:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573777.029380:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573777.605281:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573778.053335:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573778.629313:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573779.077368:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573779.653306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573780.101408:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573780.677306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573781.125397:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573781.189303:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573781.189318:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573781.189324:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573781.189337:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4929 00000100:00080000:3.0:1788573781.189351:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573781.189361:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573781.189406:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488690048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573781.189441:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573781.189446:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573781.189449:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573781.189468:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4929 00000100:00080000:3.0:1788573781.189473:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573781.189479:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573781.189485:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488690048:192.168.202.125@tcp:250:kworker.0 00000008:00100000:3.0:1788573781.189494:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573781.189515:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488690176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573781.189530:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573781.189535:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573781.189538:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573781.189543:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4929 00000100:00080000:3.0:1788573781.189551:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573781.189556:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:0.0:1788573781.189562:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:3.0:1788573781.189563:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573781.189581:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488690304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573781.189604:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573781/real 1788573781] req@ffff94d408c54700 x1875450488690048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573781.189623:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488690048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573781.189633:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573781.189637:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573781.189641:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488690048:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573781.189665:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573781.189670:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488690176:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788573781.189672:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573781.189677:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573781.189684:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4929 00000100:00080000:3.0:1788573781.189690:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788573781.189698:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573781.189698:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573781.189708:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573781.189712:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573781/real 1788573781] req@ffff94d408c55f80 x1875450488690176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573781.189723:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488690176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573781.189729:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573781.189731:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573781.189733:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488690176:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573781.189737:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488690304:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573781.189750:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573781.189757:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573781/real 1788573781] req@ffff94d408c54000 x1875450488690304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573781.189765:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488690304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573781.189775:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573781.189778:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573781.189780:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488690304:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573781.189849:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488690432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573781.189897:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488690432:192.168.202.125@tcp:8:kworker.0 00000100:00080000:3.0:1788573781.189911:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573781.189917:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573781.189920:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788573781.189922:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573781.189929:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4929 00000100:00080000:3.0:1788573781.189936:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573781.189940:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573781/real 1788573781] req@ffff94d408c56300 x1875450488690432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573781.189945:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573781.189958:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573781.189959:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488690432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573781.189969:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573781.189971:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573781.189974:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488690432:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573781.189979:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488690560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573781.190012:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488690560:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573781.190028:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573781.190037:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573781/real 1788573781] req@ffff94d408c57480 x1875450488690560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573781.190045:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488690560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573781.190050:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573781.190052:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573781.190055:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488690560:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573781.701389:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573782.149395:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573782.725290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573783.173414:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573783.749376:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573784.197394:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573784.773296:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573785.221318:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573785.797297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573786.245398:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573786.309318:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573786.309332:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573786.309337:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573786.309348:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4934 00000100:00080000:3.0:1788573786.309373:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573786.309381:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573786.309427:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488690688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573786.309457:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573786.309460:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573786.309461:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573786.309464:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4934 00000100:00080000:3.0:1788573786.309467:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573786.309470:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573786.309480:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573786.309489:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488690816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573786.309497:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573786.309500:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573786.309501:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573786.309503:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4934 00000100:00080000:3.0:1788573786.309505:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573786.309508:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573786.309512:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573786.309514:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488690688:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573786.309522:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488690944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573786.309530:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573786.309533:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573786.309534:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573786.309537:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4934 00000100:00080000:3.0:1788573786.309539:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573786.309542:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573786.309546:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573786.309562:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488691072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573786.309579:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573786.309582:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573786.309584:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573786.309589:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4934 00000800:00000100:0.0:1788573786.309594:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788573786.309595:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573786.309602:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573786.309608:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573786.309621:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488691200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573786.309638:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573786/real 1788573786] req@ffff94d408c55180 x1875450488690688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573786.309655:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488690688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573786.309664:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573786.309667:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573786.309670:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488690688:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573786.309676:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488690816:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573786.309692:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573786.309700:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573786/real 1788573786] req@ffff94d408c56a00 x1875450488690816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573786.309709:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488690816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573786.309714:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573786.309716:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573786.309718:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488690816:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573786.309742:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488690944:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573786.309753:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573786.309760:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573786/real 1788573786] req@ffff94d408c55880 x1875450488690944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573786.309768:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488690944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573786.309773:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573786.309775:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573786.309777:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488690944:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573786.309781:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488691072:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573786.309792:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573786.309798:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573786/real 1788573786] req@ffff94d408c55c00 x1875450488691072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573786.309805:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488691072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573786.309810:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573786.309812:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573786.309814:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488691072:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573786.309817:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488691200:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573786.309826:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573786.309832:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573786/real 1788573786] req@ffff94d408c57800 x1875450488691200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573786.309843:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488691200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573786.309848:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573786.309850:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573786.309852:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488691200:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573786.821331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573787.269399:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573787.845318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573788.293420:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573788.869379:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573789.317402:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573789.893348:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573790.341434:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573790.917325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573791.365437:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573791.429339:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573791.429348:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573791.429352:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573791.429367:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4939 00000100:00080000:3.0:1788573791.429378:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573791.429384:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573791.429426:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488691328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573791.429461:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573791.429465:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573791.429467:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573791.429471:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4939 00000100:00080000:3.0:1788573791.429476:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573791.429480:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573791.429491:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573791.429509:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488691456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573791.429524:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573791.429529:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573791.429530:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573791.429537:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4939 00000100:00100000:0.0:1788573791.429545:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488691328:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573791.429547:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573791.429572:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573791.429582:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573791.429603:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488691584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573791.429614:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573791.429630:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573791.429632:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573791.429636:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4939 00000100:00080000:3.0:1788573791.429641:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573791.429645:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573791.429652:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573791.429690:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573791.429698:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488691712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573791.429714:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573791.429719:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573791.429722:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573791.429730:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4939 00000100:00080000:3.0:1788573791.429737:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573791.429745:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573791/real 1788573791] req@ffff94d408c57100 x1875450488691328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573791.429747:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573791.429780:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573791.429784:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488691328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573791.429802:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788573791.429809:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488691840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573791.429812:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573791.429820:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488691328:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573791.429862:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488691456:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573791.429899:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573791.429913:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573791/real 1788573791] req@ffff94d408c54380 x1875450488691456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573791.429923:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488691456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573791.429936:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573791.429939:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573791.429942:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488691456:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573791.429948:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488691584:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573791.429961:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573791.429968:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573791/real 1788573791] req@ffff94d408c54e00 x1875450488691584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573791.429977:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488691584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573791.429984:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573791.429986:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573791.429988:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488691584:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573791.429993:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488691712:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573791.430007:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573791.430014:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573791/real 1788573791] req@ffff94d408c56d80 x1875450488691712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573791.430023:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488691712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573791.430029:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573791.430031:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573791.430034:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488691712:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573791.430038:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488691840:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573791.430050:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573791.430056:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573791/real 1788573791] req@ffff94d408c54a80 x1875450488691840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573791.430064:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488691840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573791.430070:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573791.430072:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573791.430074:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488691840:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573791.941351:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573792.389438:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573792.965300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573793.413377:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573793.989310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573794.437510:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573795.013298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573795.461360:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573796.037371:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573796.485375:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573796.549360:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573796.549371:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573796.549376:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573796.549386:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4944 00000100:00080000:1.0:1788573796.549398:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573796.549406:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573796.549452:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488691968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573796.549480:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573796.549483:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573796.549484:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573796.549487:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4944 00000100:00080000:1.0:1788573796.549489:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573796.549492:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573796.549501:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573796.549510:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488692096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573796.549522:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573796.549525:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573796.549527:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573796.549530:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4944 00000100:00080000:1.0:1788573796.549533:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573796.549535:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573796.549538:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573796.549545:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488692224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573796.549551:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573796.549555:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1788573796.549558:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488691968:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788573796.549558:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573796.549566:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4944 00000100:00080000:1.0:1788573796.549569:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573796.549571:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573796.549575:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573796.549582:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488692352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573796.549589:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573796.549591:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573796.549592:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573796.549595:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4944 00000100:00080000:1.0:1788573796.549598:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573796.549601:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573796.549605:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573796.549611:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488692480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573796.549636:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573796.549687:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573796/real 1788573796] req@ffff94d4049bfb80 x1875450488691968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573796.549704:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488691968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573796.549714:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573796.549717:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573796.549720:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488691968:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573796.549726:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488692096:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573796.549744:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573796.549753:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573796/real 1788573796] req@ffff94d4049bc000 x1875450488692096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573796.549762:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488692096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573796.549768:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573796.549770:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573796.549773:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488692096:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573796.549797:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488692224:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573796.549816:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573796.549828:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573796/real 1788573796] req@ffff94d4049bc700 x1875450488692224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573796.549842:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488692224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573796.549851:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573796.549854:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573796.549857:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488692224:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573796.549865:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488692352:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573796.549886:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573796.549898:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573796/real 1788573796] req@ffff94d4049bed80 x1875450488692352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573796.549907:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488692352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573796.549913:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573796.549916:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573796.549918:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488692352:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573796.549923:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488692480:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573796.549936:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573796.549943:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573796/real 1788573796] req@ffff94d4049bf800 x1875450488692480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573796.549951:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488692480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573796.549956:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573796.549958:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573796.549960:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488692480:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573797.061333:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573797.509399:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573798.085324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573798.533379:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573799.109295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573799.557426:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573800.133276:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573800.581374:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573801.157310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573801.605425:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573801.669316:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573801.669325:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573801.669329:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573801.669337:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4949 00000100:00080000:1.0:1788573801.669346:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573801.669353:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573801.669387:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488692608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573801.669413:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573801.669416:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573801.669418:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573801.669423:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4949 00000100:00080000:1.0:1788573801.669426:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573801.669430:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573801.669441:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573801.669454:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488692736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573801.669465:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573801.669468:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788573801.669469:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488692608:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788573801.669470:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573801.669473:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4949 00000100:00080000:1.0:1788573801.669477:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573801.669480:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573801.669484:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573801.669494:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488692864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573801.669502:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573801.669505:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573801.669506:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573801.669510:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4949 00000100:00080000:1.0:1788573801.669521:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573801.669525:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:0.0:1788573801.669530:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:1.0:1788573801.669531:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573801.669540:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488692992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573801.669549:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573801.669552:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573801.669553:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573801.669557:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4949 00000100:00080000:1.0:1788573801.669560:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573801.669563:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573801.669567:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573801/real 1788573801] req@ffff94d4049bd500 x1875450488692608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788573801.669567:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573801.669576:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488693120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573801.669595:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488692608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573801.669606:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573801.669611:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573801.669615:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488692608:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573801.669637:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488692736:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573801.669657:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573801.669667:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573801/real 1788573801] req@ffff94d4049bf100 x1875450488692736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573801.669675:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488692736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573801.669681:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573801.669683:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573801.669685:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488692736:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573801.669688:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488692864:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573801.669699:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573801.669705:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573801/real 1788573801] req@ffff94d4049bca80 x1875450488692864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573801.669712:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488692864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573801.669717:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573801.669719:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573801.669721:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488692864:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573801.669724:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488692992:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573801.669734:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573801.669739:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573801/real 1788573801] req@ffff94d4049be680 x1875450488692992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573801.669747:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488692992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573801.669752:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573801.669753:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573801.669755:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488692992:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573801.669758:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488693120:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573801.669767:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573801.669772:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573801/real 1788573801] req@ffff94d4049bdc00 x1875450488693120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573801.669778:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488693120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573801.669783:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573801.669785:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573801.669787:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488693120:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573802.181278:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573802.629360:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573803.205304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573803.653365:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573804.229279:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573804.677382:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573805.253321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573805.701454:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573806.277299:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573806.725399:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573806.789367:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573806.789380:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573806.789384:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573806.789396:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4955 00000100:00080000:3.0:1788573806.789408:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573806.789416:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573806.789461:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488693248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573806.789490:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573806.789494:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573806.789495:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573806.789498:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4955 00000100:00080000:3.0:1788573806.789505:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573806.789508:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573806.789518:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573806.789528:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488693376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573806.789541:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573806.789544:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573806.789545:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573806.789550:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4955 00000100:00100000:0.0:1788573806.789552:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488693248:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573806.789553:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573806.789560:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573806.789565:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573806.789575:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488693504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573806.789582:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573806.789584:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573806.789585:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573806.789588:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4955 00000100:00080000:3.0:1788573806.789590:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573806.789593:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573806.789596:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573806.789603:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488693632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573806.789609:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573806.789611:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573806.789612:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573806.789615:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4955 00000100:00080000:3.0:1788573806.789625:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573806.789628:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573806.789632:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573806.789637:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573806.789641:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488693760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573806.789681:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573806/real 1788573806] req@ffff94d408c56680 x1875450488693248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573806.789697:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488693248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573806.789712:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573806.789715:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573806.789726:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488693248:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573806.789751:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488693376:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573806.789769:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573806.789779:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573806/real 1788573806] req@ffff94d408c57b80 x1875450488693376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573806.789787:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488693376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573806.789793:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573806.789796:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573806.789798:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488693376:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573806.789802:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488693504:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573806.789813:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573806.789818:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573806/real 1788573806] req@ffff94d408c54a80 x1875450488693504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573806.789826:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488693504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573806.789832:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573806.789834:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573806.789835:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488693504:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573806.789839:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488693632:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573806.789851:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573806.789856:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573806/real 1788573806] req@ffff94d408c56d80 x1875450488693632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573806.789863:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488693632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573806.789869:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573806.789870:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573806.789872:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488693632:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573806.789875:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488693760:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573806.789885:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573806.789891:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573806/real 1788573806] req@ffff94d408c54e00 x1875450488693760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573806.789897:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488693760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573806.789902:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573806.789904:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573806.789906:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488693760:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573807.301311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573807.749465:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573808.325329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573808.773405:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573809.349340:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573809.797403:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573810.373331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573810.821420:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573811.397319:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573811.845419:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573811.909362:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573811.909377:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573811.909383:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573811.909397:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4960 00000100:00080000:2.0:1788573811.909411:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573811.909421:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573811.909471:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488693888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573811.909504:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573811.909509:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573811.909511:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573811.909516:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4960 00000100:00080000:2.0:1788573811.909520:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573811.909525:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573811.909538:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573811.909554:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488693888:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788573811.909555:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488694016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573811.909573:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573811.909580:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573811.909583:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573811.909587:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4960 00000100:00080000:2.0:1788573811.909592:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573811.909596:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573811.909602:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573811.909616:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488694144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573811.909628:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573811.909633:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573811.909635:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573811.909640:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4960 00000100:00080000:2.0:1788573811.909644:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788573811.909649:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788573811.909654:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573811.909662:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573811.909676:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488694272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573811.909687:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573811.909691:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573811.909694:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573811.909699:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4960 00000100:00080000:2.0:1788573811.909703:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573811.909705:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573811/real 1788573811] req@ffff94d408c63800 x1875450488693888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573811.909711:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573811.909722:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573811.909736:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488694400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573811.909737:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488693888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573811.909768:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573811.909773:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573811.909777:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488693888:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573811.909802:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488694016:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573811.909822:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573811.909832:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573811/real 1788573811] req@ffff94d408c60000 x1875450488694016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573811.909841:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488694016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573811.909847:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573811.909858:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573811.909861:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488694016:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573811.909865:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488694144:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573811.909878:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573811.909884:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573811/real 1788573811] req@ffff94d408c62d80 x1875450488694144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573811.909893:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488694144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573811.909903:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573811.909906:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573811.909908:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488694144:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573811.909911:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488694272:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573811.909940:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573811.909948:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573811/real 1788573811] req@ffff94d408c62a00 x1875450488694272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573811.909956:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488694272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573811.909961:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573811.909964:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573811.909966:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488694272:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573811.909969:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488694400:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573811.909980:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573811.909985:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573811/real 1788573811] req@ffff94d408c60a80 x1875450488694400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573811.909993:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488694400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573811.909998:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573811.910000:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573811.910002:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488694400:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573812.421353:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573812.869364:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573813.445321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573813.893383:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573814.469353:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573814.917340:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573815.493317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573815.941459:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573816.517278:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573816.965435:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573817.029347:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573817.029359:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573817.029363:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573817.029375:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4965 00000100:00080000:2.0:1788573817.029388:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573817.029406:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573817.029459:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488694528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573817.029497:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573817.029500:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573817.029502:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573817.029505:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4965 00000100:00080000:2.0:1788573817.029509:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573817.029512:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573817.029522:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573817.029535:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488694656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573817.029544:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573817.029547:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573817.029548:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573817.029554:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4965 00000100:00100000:0.0:1788573817.029555:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488694528:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788573817.029561:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573817.029565:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573817.029568:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573817.029577:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488694784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573817.029584:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573817.029587:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573817.029588:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573817.029591:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4965 00000100:00080000:2.0:1788573817.029593:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573817.029596:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573817.029599:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573817.029609:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488694912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573817.029615:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573817.029618:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573817.029619:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573817.029621:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4965 00000100:00080000:2.0:1788573817.029627:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573817.029629:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573817.029632:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573817.029639:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488695040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573817.029685:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488694656:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573817.029728:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488694784:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573817.029751:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488694912:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573817.029800:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488695040:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573817.541334:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573817.989384:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573818.565356:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573819.013462:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573819.589295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573820.037344:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788573820.101573:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:3.0:1788573820.101624:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788573820.101647:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:3.0:1788573820.101704:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:3.0:1788573820.101717:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:3.0:1788573820.101730:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:3.0:1788573820.101742:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:0.0:1788573820.101768:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573817/real 1788573820] req@ffff94d408c63b80 x1875450488694656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573820.101816:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488694656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573820.101831:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573820.101836:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573820.101842:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488694656:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573820.101852:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573817/real 1788573820] req@ffff94d408c61500 x1875450488694784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573820.101865:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488694784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573820.101875:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573820.101879:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573820.101883:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488694784:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573820.101891:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573817/real 1788573820] req@ffff94d408c63480 x1875450488694912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573820.101904:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488694912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573820.101911:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573820.101915:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573820.101919:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488694912:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573820.101926:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573817/real 1788573820] req@ffff94d408c61f80 x1875450488695040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573820.101937:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488695040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573820.101946:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573820.101949:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573820.101953:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488695040:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573820.101960:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573817/real 1788573820] req@ffff94d408c61c00 x1875450488694528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573820.101972:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488694528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573820.101979:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573820.101983:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573820.101986:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488694528:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573820.613350:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573821.061438:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573821.637304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573822.085315:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573822.149349:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573822.149358:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573822.149361:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573822.149370:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4970 00000100:00080000:3.0:1788573822.149380:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573822.149386:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573822.149415:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488695168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573822.149440:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573822.149443:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573822.149445:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573822.149448:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4970 00000100:00080000:3.0:1788573822.149452:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573822.149456:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573822.149471:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573822.149487:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488695296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573822.149492:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488695168:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573822.149508:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573822.149512:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573822.149515:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573822.149520:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4970 00000100:00080000:3.0:1788573822.149525:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573822.149530:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573822.149536:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788573822.149558:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573822.149560:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488695424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573822.149571:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573822.149575:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573822.149578:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573822.149583:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4970 00000100:00080000:3.0:1788573822.149588:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573822.149596:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573822.149597:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573822/real 1788573822] req@ffff94d408c54380 x1875450488695168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788573822.149604:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573822.149623:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488695552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573822.149628:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488695168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573822.149659:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573822.149668:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:3.0:1788573822.149669:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573822.149677:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573822.149679:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573822.149685:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4970 00000100:00100000:0.0:1788573822.149690:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488695168:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788573822.149696:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573822.149703:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788573822.149708:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573822.149722:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488695680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573822.149726:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488695296:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573822.149777:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573822.149796:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573822/real 1788573822] req@ffff94d408c57100 x1875450488695296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573822.149818:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488695296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573822.149827:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573822.149831:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573822.149834:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488695296:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573822.149841:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488695424:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573822.149864:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573822.149877:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573822/real 1788573822] req@ffff94d408c57800 x1875450488695424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573822.149896:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488695424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573822.149905:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573822.149907:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573822.149911:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488695424:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573822.149917:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488695552:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573822.149939:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573822.149950:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573822/real 1788573822] req@ffff94d408c55c00 x1875450488695552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573822.149970:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488695552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573822.149979:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573822.149983:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573822.149986:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488695552:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573822.150004:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488695680:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573822.150035:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573822.150048:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573822/real 1788573822] req@ffff94d408c55880 x1875450488695680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573822.150061:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488695680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573822.150075:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573822.150078:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573822.150082:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488695680:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573822.661309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573823.109364:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573823.685316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573824.133425:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573824.709285:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573825.157420:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573825.733954:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573826.181395:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573826.757289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573827.205344:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573827.269323:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573827.269335:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573827.269340:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573827.269351:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4975 00000100:00080000:0.0:1788573827.269364:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573827.269372:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573827.269436:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488695808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573827.269470:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573827.269475:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573827.269477:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573827.269482:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4975 00000100:00080000:0.0:1788573827.269487:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573827.269493:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573827.269505:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573827.269505:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488695808:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573827.269522:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488695936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573827.269532:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573827.269535:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573827.269538:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573827.269541:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4975 00000100:00080000:0.0:1788573827.269544:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573827.269547:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573827.269551:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573827.269560:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488696064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573827.269567:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573827.269570:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573827.269571:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573827.269573:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4975 00000100:00080000:0.0:1788573827.269576:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573827.269578:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573827.269582:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788573827.269587:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573827.269592:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488696192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573827.269599:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573827.269602:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573827.269603:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573827.269606:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4975 00000100:00080000:0.0:1788573827.269609:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573827.269611:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573827.269614:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573827.269621:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488696320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573827.269644:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573827/real 1788573827] req@ffff94d408d3dc00 x1875450488695808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573827.269701:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488695808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573827.269711:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573827.269714:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573827.269717:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488695808:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573827.269742:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488695936:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573827.269762:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573827.269773:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573827/real 1788573827] req@ffff94d408d3c380 x1875450488695936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573827.269782:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488695936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573827.269788:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573827.269790:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573827.269792:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488695936:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573827.269796:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488696064:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573827.269807:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573827.269814:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573827/real 1788573827] req@ffff94d408d3f480 x1875450488696064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573827.269821:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488696064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573827.269827:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573827.269828:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573827.269840:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488696064:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573827.269847:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488696192:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573827.269861:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573827.269869:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573827/real 1788573827] req@ffff94d408d3c700 x1875450488696192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573827.269882:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488696192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573827.269891:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573827.269895:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573827.269899:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488696192:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573827.269905:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488696320:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573827.269924:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573827.269932:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573827/real 1788573827] req@ffff94d408d3e300 x1875450488696320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573827.269941:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488696320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573827.269947:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573827.269949:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573827.269951:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488696320:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573827.781298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573828.229450:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573828.805279:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573829.253380:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573829.829313:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573830.277383:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573830.853798:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573831.301429:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573831.877274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573832.325391:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573832.389293:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573832.389301:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573832.389305:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573832.389313:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4980 00000100:00080000:0.0:1788573832.389323:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573832.389329:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573832.389363:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488696448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573832.389391:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573832.389394:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573832.389396:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573832.389398:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4980 00000100:00080000:0.0:1788573832.389401:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573832.389403:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573832.389410:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573832.389419:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ed80 x1875450488696576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573832.389427:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573832.389429:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573832.389431:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573832.389434:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4980 00000100:00080000:0.0:1788573832.389440:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573832.389443:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573832.389446:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573832.389454:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488696704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573832.389462:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573832.389464:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573832.389465:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573832.389468:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4980 00000100:00080000:0.0:1788573832.389470:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573832.389475:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573832.389480:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573832.389486:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488696448:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573832.389489:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488696832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573832.389509:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573832.389511:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573832.389512:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573832.389517:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4980 00000100:00080000:0.0:1788573832.389519:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573832.389524:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573832.389529:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573832.389539:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488696960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573832.389607:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573832.389664:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573832/real 1788573832] req@ffff94d408d3d880 x1875450488696448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573832.389684:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488696448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573832.389694:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573832.389698:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573832.389701:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488696448:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573832.389708:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488696576:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573832.389726:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573832.389736:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573832/real 1788573832] req@ffff94d408d3ed80 x1875450488696576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573832.389744:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ed80 x1875450488696576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573832.389750:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573832.389753:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573832.389755:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488696576:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573832.389780:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488696704:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573832.389791:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573832.389797:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573832/real 1788573832] req@ffff94d408d3e300 x1875450488696704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573832.389805:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488696704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573832.389811:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573832.389813:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573832.389815:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488696704:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573832.389820:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488696832:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573832.389832:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573832.389838:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573832/real 1788573832] req@ffff94d408d3c700 x1875450488696832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573832.389846:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488696832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573832.389851:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573832.389853:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573832.389854:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488696832:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573832.389858:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488696960:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573832.389868:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573832.389873:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573832/real 1788573832] req@ffff94d408d3f480 x1875450488696960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573832.389880:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488696960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573832.389885:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573832.389888:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573832.389889:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488696960:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573832.901315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573833.349426:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573833.925293:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573834.373404:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573834.949317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573835.397420:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573835.973257:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573836.421379:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573836.997315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573837.445397:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573837.509361:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573837.509370:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573837.509373:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573837.509383:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4985 00000100:00080000:0.0:1788573837.509394:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573837.509402:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573837.509440:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488697088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573837.509499:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573837.509503:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573837.509504:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573837.509508:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4985 00000100:00080000:0.0:1788573837.509511:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573837.509514:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573837.509523:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573837.509535:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488697216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573837.509544:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573837.509548:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573837.509552:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573837.509556:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488697088:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788573837.509557:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4985 00000100:00080000:0.0:1788573837.509562:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573837.509570:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573837.509575:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573837.509582:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488697344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573837.509589:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573837.509592:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573837.509593:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573837.509596:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4985 00000100:00080000:0.0:1788573837.509601:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573837.509603:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573837.509606:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573837.509613:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488697472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573837.509619:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573837.509622:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573837.509623:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573837.509626:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4985 00000100:00080000:0.0:1788573837.509629:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573837.509631:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573837.509635:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573837.509643:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488697600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573837.509669:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573837.509720:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573837/real 1788573837] req@ffff94d408d3c380 x1875450488697088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573837.509743:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488697088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573837.509757:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573837.509762:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573837.509766:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488697088:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573837.509799:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488697216:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573837.509823:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573837.509836:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573837/real 1788573837] req@ffff94d408d3dc00 x1875450488697216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573837.509850:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488697216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573837.509859:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573837.509863:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573837.509867:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488697216:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573837.509875:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488697344:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573837.509909:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573837.509922:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573837/real 1788573837] req@ffff94d408d3c000 x1875450488697344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573837.509935:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488697344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573837.509944:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573837.509948:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573837.509951:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488697344:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573837.509958:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488697472:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573837.509977:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573837.509989:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573837/real 1788573837] req@ffff94d408d3d180 x1875450488697472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573837.510002:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488697472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573837.510011:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573837.510015:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573837.510018:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488697472:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573837.510025:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488697600:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573837.510045:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573837.510057:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573837/real 1788573837] req@ffff94d408d3df80 x1875450488697600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573837.510070:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488697600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573837.510086:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573837.510088:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573837.510091:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488697600:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573838.021318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573838.469396:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573839.045319:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573839.493406:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573840.069308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573840.517476:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573841.093321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573841.541388:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573842.117302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573842.565444:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573842.629309:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573842.629321:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573842.629326:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573842.629337:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4990 00000100:00080000:1.0:1788573842.629349:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573842.629357:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573842.629424:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd880 x1875450488697728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573842.629446:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573842.629449:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573842.629451:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573842.629454:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4990 00000100:00080000:1.0:1788573842.629457:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573842.629460:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573842.629467:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573842.629478:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488697856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573842.629486:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573842.629488:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573842.629491:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573842.629494:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488697728:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788573842.629495:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4990 00000100:00080000:1.0:1788573842.629500:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573842.629505:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573842.629508:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573842.629517:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488697984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573842.629527:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573842.629530:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573842.629533:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573842.629537:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4990 00000100:00080000:1.0:1788573842.629541:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573842.629545:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573842.629551:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788573842.629561:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573842.629568:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488698112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573842.629580:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573842.629583:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573842.629585:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573842.629590:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4990 00000100:00080000:1.0:1788573842.629594:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573842.629598:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573842.629602:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573842.629604:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573842/real 1788573842] req@ffff94d4049bd880 x1875450488697728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573842.629614:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488698240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573842.629636:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd880 x1875450488697728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573842.629649:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573842.629654:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573842.629658:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488697728:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573842.629681:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488697856:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573842.629703:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573842.629715:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573842/real 1788573842] req@ffff94d4049be300 x1875450488697856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573842.629727:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488697856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573842.629736:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573842.629740:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573842.629743:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488697856:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573842.629749:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488697984:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573842.629763:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573842.629772:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573842/real 1788573842] req@ffff94d4049bdc00 x1875450488697984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573842.629783:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488697984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573842.629792:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573842.629795:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573842.629797:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488697984:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573842.629801:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488698112:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573842.629815:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573842.629822:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573842/real 1788573842] req@ffff94d4049be680 x1875450488698112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573842.629830:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488698112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573842.629835:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573842.629837:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573842.629840:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488698112:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573842.629844:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488698240:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573842.629853:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573842.629858:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573842/real 1788573842] req@ffff94d4049bca80 x1875450488698240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573842.629865:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488698240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573842.629870:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573842.629872:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573842.629874:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488698240:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573843.141392:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573843.589435:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573844.165349:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573844.613432:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573845.189310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573845.637455:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573846.213277:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573846.661455:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573847.237321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573847.685465:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573847.749338:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573847.749347:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573847.749351:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573847.749359:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 4996 00000100:00080000:1.0:1788573847.749369:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573847.749378:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573847.749417:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488698368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573847.749445:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573847.749447:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573847.749449:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573847.749452:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4996 00000100:00080000:1.0:1788573847.749455:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573847.749459:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573847.749467:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573847.749479:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488698496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573847.749488:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573847.749490:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573847.749492:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573847.749495:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4996 00000100:00080000:1.0:1788573847.749499:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573847.749502:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573847.749507:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573847.749517:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488698368:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573847.749518:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488698624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573847.749542:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573847.749544:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573847.749545:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573847.749549:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4996 00000100:00080000:1.0:1788573847.749551:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573847.749554:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573847.749558:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573847.749567:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488698752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573847.749573:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573847.749576:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573847.749577:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573847.749581:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 4996 00000100:00080000:1.0:1788573847.749586:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573847.749590:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573847.749593:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573847.749599:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488698880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573847.749614:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573847.749656:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573847/real 1788573847] req@ffff94d4049bf100 x1875450488698368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573847.749674:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488698368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573847.749683:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573847.749687:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573847.749690:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488698368:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573847.749697:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488698496:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573847.749713:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573847.749721:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573847/real 1788573847] req@ffff94d4049bd500 x1875450488698496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573847.749730:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488698496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573847.749736:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573847.749738:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573847.749741:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488698496:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573847.749764:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488698624:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573847.749775:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573847.749781:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573847/real 1788573847] req@ffff94d4049bf800 x1875450488698624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573847.749789:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488698624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573847.749795:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573847.749797:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573847.749799:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488698624:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573847.749803:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488698752:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573847.749814:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573847.749820:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573847/real 1788573847] req@ffff94d4049bed80 x1875450488698752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573847.749828:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488698752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573847.749846:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573847.749853:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573847.749857:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488698752:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573847.749864:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488698880:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573847.749886:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573847.749899:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573847/real 1788573847] req@ffff94d4049bc700 x1875450488698880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573847.749911:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488698880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573847.749919:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573847.749922:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573847.749925:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488698880:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573848.261355:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573848.709458:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573849.285370:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573849.733413:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573850.309375:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573850.757470:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573851.333362:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573851.781403:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573852.357348:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573852.805411:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573852.869382:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573852.869397:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573852.869403:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573852.869417:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5001 00000100:00080000:1.0:1788573852.869434:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573852.869444:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573852.869498:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488699008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573852.869539:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573852.869543:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573852.869546:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573852.869550:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5001 00000100:00080000:1.0:1788573852.869554:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573852.869559:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573852.869571:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573852.869588:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488699136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573852.869602:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573852.869609:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788573852.869610:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488699008:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788573852.869612:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573852.869622:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5001 00000100:00080000:1.0:1788573852.869627:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573852.869631:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573852.869637:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573852.869668:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488699264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573852.869679:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573852.869683:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573852.869686:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573852.869691:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5001 00000100:00080000:1.0:1788573852.869696:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788573852.869703:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788573852.869704:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573852.869714:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573852.869730:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488699392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573852.869742:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573852.869746:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573852.869748:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573852.869754:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5001 00000100:00080000:1.0:1788573852.869758:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788573852.869761:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573852/real 1788573852] req@ffff94d4049bc000 x1875450488699008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573852.869767:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573852.869778:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573852.869793:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488699520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573852.869797:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488699008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573852.869823:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573852.869829:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573852.869834:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488699008:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573852.869867:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488699136:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573852.869898:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573852.869916:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573852/real 1788573852] req@ffff94d4049bfb80 x1875450488699136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573852.869932:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488699136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573852.869944:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573852.869949:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573852.869953:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488699136:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573852.869962:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488699264:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573852.870004:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573852.870025:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573852/real 1788573852] req@ffff94d4049bea00 x1875450488699264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573852.870041:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488699264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573852.870059:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573852.870064:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573852.870069:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488699264:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573852.870078:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488699392:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573852.870106:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573852.870120:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573852/real 1788573852] req@ffff94d4049bf480 x1875450488699392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573852.870137:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488699392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573852.870146:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573852.870150:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573852.870153:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488699392:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573852.870162:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488699520:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573852.870459:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573852.870525:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573852/real 1788573852] req@ffff94d4049bc380 x1875450488699520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573852.870542:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc380 x1875450488699520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573852.870551:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573852.870554:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573852.870559:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488699520:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573853.381293:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573853.829384:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573854.405303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573854.853376:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573855.429286:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573855.877435:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573856.453308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573856.901368:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573857.477272:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573857.925371:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573857.989310:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573857.989320:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573857.989324:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573857.989335:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5006 00000100:00080000:2.0:1788573857.989345:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573857.989353:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573857.989390:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488699648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573857.989412:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573857.989415:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573857.989416:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573857.989419:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5006 00000100:00080000:2.0:1788573857.989422:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573857.989425:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573857.989433:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573857.989443:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63100 x1875450488699776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573857.989450:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488699648:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788573857.989453:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573857.989459:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573857.989461:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573857.989465:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5006 00000100:00080000:2.0:1788573857.989467:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573857.989469:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573857.989473:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573857.989481:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60e00 x1875450488699904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573857.989488:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573857.989490:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573857.989492:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573857.989494:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5006 00000100:00080000:2.0:1788573857.989503:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573857.989506:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573857.989510:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788573857.989513:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573857.989522:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488700032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573857.989530:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573857.989532:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573857.989533:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573857.989536:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5006 00000100:00080000:2.0:1788573857.989539:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573857.989542:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573857.989546:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573857.989547:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573857/real 1788573857] req@ffff94d408c61880 x1875450488699648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573857.989556:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488700160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573857.989573:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488699648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573857.989582:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573857.989586:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573857.989589:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488699648:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573857.989607:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488699776:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573857.989623:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573857.989631:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573857/real 1788573857] req@ffff94d408c63100 x1875450488699776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573857.989639:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63100 x1875450488699776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573857.989645:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573857.989648:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573857.989651:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488699776:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573857.989656:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488699904:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573857.989675:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573857.989687:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573857/real 1788573857] req@ffff94d408c60e00 x1875450488699904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573857.989701:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60e00 x1875450488699904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573857.989710:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573857.989714:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573857.989716:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488699904:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573857.989721:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488700032:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573857.989737:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573857.989744:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573857/real 1788573857] req@ffff94d408c61c00 x1875450488700032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573857.989753:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488700032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573857.989758:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573857.989760:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573857.989763:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488700032:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573857.989766:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488700160:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573857.989776:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573857.989782:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573857/real 1788573857] req@ffff94d408c61f80 x1875450488700160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573857.989789:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488700160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573857.989794:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573857.989796:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573857.989798:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488700160:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573858.501298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573858.949338:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573859.525308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573859.973369:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573860.549301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573860.997342:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573861.573354:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573862.021420:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573862.597290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573863.045377:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573863.109355:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573863.109366:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573863.109371:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573863.109383:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5011 00000100:00080000:0.0:1788573863.109395:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573863.109403:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573863.109447:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488700288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573863.109482:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573863.109487:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573863.109489:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573863.109494:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5011 00000100:00080000:0.0:1788573863.109499:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573863.109504:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573863.109515:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573863.109536:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488700416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573863.109546:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573863.109549:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573863.109551:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573863.109554:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5011 00000100:00080000:0.0:1788573863.109558:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788573863.109561:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488700288:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788573863.109563:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573863.109571:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573863.109579:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488700544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573863.109586:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573863.109588:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573863.109590:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573863.109593:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5011 00000100:00080000:0.0:1788573863.109595:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573863.109597:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573863.109601:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573863.109610:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488700672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573863.109616:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573863.109619:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573863.109620:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573863.109622:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5011 00000100:00080000:0.0:1788573863.109624:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573863.109627:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573863.109630:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573863.109636:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488700800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573863.109647:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573863.109687:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573863/real 1788573863] req@ffff94d408d3ce00 x1875450488700288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573863.109703:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488700288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573863.109713:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573863.109716:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573863.109720:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488700288:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573863.109743:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488700416:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573863.109758:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573863.109767:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573863/real 1788573863] req@ffff94d408d3ea00 x1875450488700416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573863.109775:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488700416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573863.109781:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573863.109783:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573863.109785:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488700416:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573863.109789:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488700544:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573863.109798:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573863.109804:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573863/real 1788573863] req@ffff94d408d3e680 x1875450488700544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573863.109813:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488700544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573863.109821:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573863.109825:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573863.109828:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488700544:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573863.109834:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488700672:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573863.109857:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573863.109869:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573863/real 1788573863] req@ffff94d408d3d500 x1875450488700672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573863.109878:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488700672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573863.109883:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573863.109885:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573863.109887:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488700672:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573863.109891:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488700800:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573863.109905:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573863.109911:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573863/real 1788573863] req@ffff94d408d3ca80 x1875450488700800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573863.109918:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488700800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573863.109923:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573863.109925:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573863.109927:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488700800:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573863.621292:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573864.069429:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573864.645286:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573865.093361:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573865.669351:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573866.117438:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573866.693272:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573867.141413:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573867.717338:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573868.165394:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573868.229393:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573868.229404:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573868.229409:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573868.229420:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5016 00000100:00080000:1.0:1788573868.229433:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573868.229441:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573868.229488:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488700928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573868.229517:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573868.229520:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573868.229523:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573868.229526:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5016 00000100:00080000:1.0:1788573868.229528:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573868.229531:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573868.229541:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573868.229551:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488701056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573868.229559:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573868.229562:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573868.229563:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573868.229567:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5016 00000100:00080000:1.0:1788573868.229569:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573868.229573:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573868.229578:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573868.229578:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488700928:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573868.229590:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488701184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573868.229597:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573868.229600:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573868.229602:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573868.229605:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5016 00000100:00080000:1.0:1788573868.229607:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573868.229610:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573868.229614:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573868.229623:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488701312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573868.229630:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573868.229632:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573868.229634:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573868.229636:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5016 00000100:00080000:1.0:1788573868.229639:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573868.229641:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573868.229643:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573868.229649:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488701440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573868.229669:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573868.229716:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573868/real 1788573868] req@ffff94d4049bce00 x1875450488700928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573868.229739:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488700928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573868.229753:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573868.229758:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573868.229763:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488700928:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573868.229771:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488701056:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573868.229793:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573868.229805:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573868/real 1788573868] req@ffff94d4049bd180 x1875450488701056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573868.229819:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488701056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573868.229830:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573868.229833:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573868.229837:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488701056:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573868.229866:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488701184:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573868.229904:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573868.229917:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573868/real 1788573868] req@ffff94d4049bc380 x1875450488701184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573868.229930:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc380 x1875450488701184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573868.229940:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573868.229944:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573868.229948:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488701184:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573868.229968:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488701312:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573868.229989:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573868.230000:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573868/real 1788573868] req@ffff94d4049bf480 x1875450488701312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573868.230013:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488701312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573868.230022:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573868.230026:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573868.230029:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488701312:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573868.230036:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488701440:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573868.230054:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573868.230064:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573868/real 1788573868] req@ffff94d4049bea00 x1875450488701440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573868.230077:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488701440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573868.230086:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573868.230090:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573868.230093:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488701440:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573868.741297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573869.189325:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573869.765350:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573870.213403:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573870.789369:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573871.237370:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573871.813342:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573872.261394:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573872.837296:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573873.285353:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573873.349291:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573873.349306:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573873.349312:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573873.349325:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5021 00000100:00080000:1.0:1788573873.349340:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573873.349351:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573873.349395:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488701568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573873.349424:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573873.349427:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573873.349430:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573873.349433:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5021 00000100:00080000:1.0:1788573873.349436:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573873.349447:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573873.349457:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573873.349467:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488701568:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573873.349469:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488701696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573873.349481:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573873.349484:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573873.349485:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573873.349488:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5021 00000100:00080000:1.0:1788573873.349491:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573873.349493:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573873.349497:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573873.349506:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488701824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573873.349513:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573873.349516:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573873.349517:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573873.349520:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5021 00000100:00080000:1.0:1788573873.349523:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573873.349525:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573873.349529:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573873.349536:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488701952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573873.349543:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573873.349546:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573873.349547:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573873.349549:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5021 00000100:00080000:1.0:1788573873.349551:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573873.349555:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:3.0:1788573873.349558:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:1.0:1788573873.349560:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573873.349569:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488702080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573873.349610:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573873/real 1788573873] req@ffff94d4049bfb80 x1875450488701568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573873.349629:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488701568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573873.349639:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573873.349642:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573873.349646:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488701568:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573873.349669:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488701696:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573873.349690:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573873.349698:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573873/real 1788573873] req@ffff94d4049bc000 x1875450488701696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573873.349707:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488701696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573873.349712:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573873.349715:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573873.349717:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488701696:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573873.349721:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488701824:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573873.349731:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573873.349737:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573873/real 1788573873] req@ffff94d4049bc700 x1875450488701824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573873.349745:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488701824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573873.349750:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573873.349752:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573873.349754:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488701824:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573873.349757:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488701952:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573873.349789:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573873.349797:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573873/real 1788573873] req@ffff94d4049bed80 x1875450488701952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573873.349805:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488701952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573873.349811:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573873.349813:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573873.349815:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488701952:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573873.349819:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488702080:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573873.349833:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573873.349843:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573873/real 1788573873] req@ffff94d4049bf800 x1875450488702080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573873.349855:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488702080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573873.349870:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573873.349874:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573873.349877:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488702080:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573873.861298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573874.309406:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573874.885287:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573875.333361:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573875.909338:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573876.357394:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573876.933369:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573877.381438:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573877.957332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573878.405362:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573878.469358:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573878.469369:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573878.469374:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573878.469386:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5026 00000100:00080000:0.0:1788573878.469401:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573878.469412:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573878.469464:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f100 x1875450488702208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573878.469509:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573878.469514:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573878.469517:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573878.469524:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5026 00000100:00080000:0.0:1788573878.469529:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573878.469536:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573878.469549:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573878.469563:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488702208:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573878.469570:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f800 x1875450488702336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573878.469586:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573878.469592:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573878.469595:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573878.469608:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5026 00000100:00080000:0.0:1788573878.469615:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573878.469622:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573878.469630:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788573878.469636:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573878.469654:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488702464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573878.469680:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573878.469683:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573878.469684:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573878.469688:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5026 00000100:00080000:0.0:1788573878.469692:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573878.469696:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573878.469701:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573878.469705:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573878/real 1788573878] req@ffff94d408d3f100 x1875450488702208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573878.469715:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488702592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573878.469726:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573878.469732:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1788573878.469733:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f100 x1875450488702208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573878.469740:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573878.469752:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5026 00000100:00080000:3.0:1788573878.469768:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573878.469773:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:0.0:1788573878.469774:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788573878.469777:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488702208:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788573878.469790:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573878.469798:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573878.469804:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488702336:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573878.469814:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488702720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573878.469831:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573878.469851:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573878/real 1788573878] req@ffff94d408d3f800 x1875450488702336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573878.469867:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f800 x1875450488702336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573878.469878:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573878.469883:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573878.469887:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488702336:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573878.469895:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488702464:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573878.469919:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573878.469934:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573878/real 1788573878] req@ffff94d408d3ca80 x1875450488702464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573878.469948:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488702464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573878.469958:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573878.469962:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573878.469966:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488702464:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573878.469973:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488702592:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573878.469995:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573878.470006:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573878/real 1788573878] req@ffff94d408d3d500 x1875450488702592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573878.470019:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488702592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573878.470028:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573878.470031:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573878.470035:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488702592:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573878.470052:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488702720:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573878.470068:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573878.470076:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573878/real 1788573878] req@ffff94d408d3e680 x1875450488702720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573878.470085:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488702720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573878.470090:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573878.470092:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573878.470094:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488702720:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573878.981353:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573879.429450:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573880.005298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573880.453368:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573881.029380:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573881.477371:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573882.053368:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573882.501418:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573883.077309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573883.525371:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573883.589286:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573883.589298:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573883.589302:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573883.589312:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5031 00000100:00080000:1.0:1788573883.589324:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573883.589330:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573883.589363:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488702848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573883.589389:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573883.589401:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573883.589403:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573883.589408:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5031 00000100:00080000:1.0:1788573883.589412:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573883.589418:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573883.589429:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573883.589431:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488702848:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573883.589447:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488702976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573883.589458:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573883.589461:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573883.589463:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573883.589466:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5031 00000100:00080000:1.0:1788573883.589470:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573883.589472:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573883.589476:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573883.589488:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488703104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573883.589498:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573883.589502:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573883.589504:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573883.589509:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5031 00000100:00080000:1.0:1788573883.589514:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788573883.589519:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488702976:192.168.202.125@tcp:38:kworker.0 00000100:00080000:1.0:1788573883.589522:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573883.589532:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573883.589547:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488703232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573883.589554:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488703104:192.168.202.125@tcp:38:kworker.0 00000100:00080000:1.0:1788573883.589578:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573883.589582:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573883.589584:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573883.589588:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5031 00000100:00080000:1.0:1788573883.589592:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573883.589595:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573883.589600:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573883.589608:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488703232:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788573883.589613:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488703360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573883.589654:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488703360:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573884.101305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573884.549419:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573885.125314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573885.573373:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573886.149343:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573886.597326:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788573886.661463:0:2392:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:0.0:1788573886.661514:0:2392:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788573886.661530:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:0.0:1788573886.661574:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:0.0:1788573886.661594:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:0.0:1788573886.661606:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:0.0:1788573886.661619:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:3.0:1788573886.661632:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573883/real 1788573886] req@ffff94d4049be680 x1875450488703232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573886.661660:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488703232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573886.661672:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573886.661677:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573886.661682:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488703232:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573886.661690:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573883/real 1788573886] req@ffff94d4049bf100 x1875450488702976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573886.661702:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488702976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573886.661710:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573886.661714:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573886.661717:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488702976:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573886.661724:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573883/real 1788573886] req@ffff94d4049bca80 x1875450488703104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573886.661735:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488703104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573886.661744:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573886.661747:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573886.661751:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488703104:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573886.661758:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573883/real 1788573886] req@ffff94d4049bd500 x1875450488702848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573886.661770:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488702848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573886.661778:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573886.661782:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573886.661785:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488702848:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573886.661812:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573883/real 1788573886] req@ffff94d4049bdc00 x1875450488703360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573886.661823:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488703360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573886.661831:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573886.661834:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573886.661838:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488703360:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573887.173297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573887.621396:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573888.197291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573888.645345:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573888.709321:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573888.709336:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573888.709342:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573888.709357:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5037 00000100:00080000:2.0:1788573888.709372:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573888.709383:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573888.709447:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488703488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573888.709494:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573888.709499:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573888.709501:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573888.709507:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5037 00000100:00080000:2.0:1788573888.709512:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573888.709519:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573888.709532:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573888.709543:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488703488:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788573888.709553:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488703616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573888.709569:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573888.709573:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573888.709576:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573888.709583:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5037 00000100:00080000:2.0:1788573888.709590:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573888.709598:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573888.709606:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573888.709622:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488703744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573888.709640:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788573888.709645:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573888.709650:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573888.709653:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573888.709660:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5037 00000100:00080000:2.0:1788573888.709666:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573888.709673:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573888.709680:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573888.709702:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488703872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573888.709712:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573888/real 1788573888] req@ffff94d408c63480 x1875450488703488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573888.709731:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573888.709738:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573888.709742:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573888.709748:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488703488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573888.709750:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5037 00000100:00080000:2.0:1788573888.709764:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573888.709770:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573888.709773:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00080000:3.0:1788573888.709780:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000008:00100000:2.0:1788573888.709782:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573888.709789:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488703488:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788573888.709802:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488704000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573888.709842:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488703616:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573888.709878:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573888.709894:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573888/real 1788573888] req@ffff94d408c61500 x1875450488703616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573888.709908:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488703616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573888.709918:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573888.709922:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573888.709925:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488703616:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573888.709933:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488703744:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573888.709954:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573888.709966:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573888/real 1788573888] req@ffff94d408c63b80 x1875450488703744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573888.709979:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488703744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573888.709995:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573888.709999:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573888.710003:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488703744:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573888.710011:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488703872:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573888.710053:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573888.710068:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573888/real 1788573888] req@ffff94d408c60a80 x1875450488703872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573888.710080:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488703872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573888.710332:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573888.710338:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573888.710342:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488703872:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573888.710350:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488704000:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573888.710377:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573888.710391:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573888/real 1788573888] req@ffff94d408c62a00 x1875450488704000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573888.710404:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488704000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573888.710412:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573888.710416:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573888.710420:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488704000:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573889.221278:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573889.669460:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573890.245323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573890.693370:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573891.269312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573891.717444:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573892.293474:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573892.741412:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573893.317569:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573893.765351:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573893.829328:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573893.829338:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573893.829343:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573893.829368:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5042 00000100:00080000:1.0:1788573893.829380:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573893.829388:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573893.829444:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488704128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573893.829490:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573893.829495:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573893.829497:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573893.829502:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5042 00000100:00080000:1.0:1788573893.829507:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573893.829521:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573893.829531:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573893.829546:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488704128:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573893.829566:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd880 x1875450488704256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573893.829584:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573893.829589:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573893.829591:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573893.829598:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5042 00000100:00080000:1.0:1788573893.829604:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573893.829609:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573893.829616:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788573893.829633:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573893.829636:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488704384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573893.829652:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573893.829658:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573893.829660:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573893.829666:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5042 00000100:00080000:1.0:1788573893.829671:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573893.829677:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573893.829680:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573893/real 1788573893] req@ffff94d4049be300 x1875450488704128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788573893.829690:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573893.829709:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488704128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573893.829715:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488704512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573893.829734:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573893.829743:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:1.0:1788573893.829744:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788573893.829750:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488704128:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788573893.829752:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573893.829761:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573893.829771:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5042 00000100:00080000:1.0:1788573893.829785:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788573893.829790:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488704256:192.168.202.125@tcp:38:kworker.0 00000100:00080000:1.0:1788573893.829799:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573893.829813:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788573893.829827:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573893.829834:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488704640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573893.829842:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573893/real 1788573893] req@ffff94d4049bd880 x1875450488704256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573893.829892:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd880 x1875450488704256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573893.829904:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573893.829916:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573893.829922:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488704256:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573893.829931:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488704384:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573893.829956:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573893.829967:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573893/real 1788573893] req@ffff94d4049bdc00 x1875450488704384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573893.829984:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488704384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573893.829991:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573893.829994:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573893.829996:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488704384:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573893.830001:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488704512:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573893.830016:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573893.830024:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573893/real 1788573893] req@ffff94d4049bd500 x1875450488704512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573893.830032:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488704512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573893.830037:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573893.830039:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573893.830041:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488704512:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573893.830059:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488704640:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573893.830075:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573893.830082:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573893/real 1788573893] req@ffff94d4049bca80 x1875450488704640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573893.830090:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488704640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573893.830100:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573893.830102:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573893.830104:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488704640:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573894.341557:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573894.789421:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573895.365590:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573895.813429:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573896.389329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573896.837416:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573897.413347:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573897.861333:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573898.437376:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573898.885386:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573898.949314:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573898.949323:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573898.949327:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573898.949335:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5047 00000100:00080000:1.0:1788573898.949346:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573898.949353:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573898.949387:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488704768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573898.949407:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573898.949410:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573898.949411:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573898.949414:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5047 00000100:00080000:1.0:1788573898.949416:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573898.949418:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573898.949426:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573898.949435:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488704896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573898.949443:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573898.949445:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573898.949446:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573898.949451:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5047 00000100:00100000:3.0:1788573898.949452:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488704768:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788573898.949453:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573898.949460:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573898.949464:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573898.949472:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488705024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573898.949478:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573898.949481:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573898.949483:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573898.949485:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5047 00000100:00080000:1.0:1788573898.949488:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573898.949491:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573898.949494:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573898.949501:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488705152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573898.949507:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573898.949509:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573898.949510:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788573898.949515:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788573898.949516:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5047 00000100:00080000:1.0:1788573898.949521:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573898.949523:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573898.949527:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573898.949535:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488705280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573898.949555:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573898/real 1788573898] req@ffff94d4049bf100 x1875450488704768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573898.949571:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488704768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573898.949578:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573898.949582:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573898.949585:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488704768:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573898.949601:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488704896:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573898.949629:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573898.949637:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573898/real 1788573898] req@ffff94d4049be680 x1875450488704896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573898.949650:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488704896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573898.949656:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573898.949658:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573898.949660:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488704896:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573898.949664:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488705024:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573898.949675:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573898.949681:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573898/real 1788573898] req@ffff94d4049bf800 x1875450488705024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573898.949689:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488705024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573898.949694:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573898.949696:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573898.949698:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488705024:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573898.949701:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488705152:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573898.949711:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573898.949716:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573898/real 1788573898] req@ffff94d4049bed80 x1875450488705152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573898.949723:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488705152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573898.949733:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573898.949735:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573898.949737:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488705152:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573898.949740:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488705280:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573898.949749:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573898.949754:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573898/real 1788573898] req@ffff94d4049bc700 x1875450488705280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573898.949761:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488705280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573898.949766:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573898.949767:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573898.949769:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488705280:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573899.461287:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573899.909343:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573900.485322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573900.933391:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573901.509260:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573901.957375:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573902.533291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573902.981433:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573903.557293:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573904.005433:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573904.069339:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573904.069355:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573904.069360:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573904.069375:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5052 00000100:00080000:1.0:1788573904.069390:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573904.069400:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573904.069453:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488705408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573904.069502:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573904.069507:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573904.069509:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573904.069514:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5052 00000100:00080000:1.0:1788573904.069519:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573904.069524:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573904.069536:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573904.069552:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488705536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573904.069565:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788573904.069566:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488705408:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788573904.069576:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573904.069580:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573904.069585:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5052 00000100:00080000:1.0:1788573904.069589:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573904.069594:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573904.069599:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573904.069614:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488705664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573904.069625:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573904.069629:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573904.069631:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573904.069635:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5052 00000100:00080000:1.0:1788573904.069641:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573904.069646:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573904.069652:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788573904.069658:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573904.069670:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488705792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573904.069683:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573904.069687:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573904.069689:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573904.069694:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5052 00000100:00080000:1.0:1788573904.069699:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573904.069704:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573904.069710:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573904.069710:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573904/real 1788573904] req@ffff94d4049bc000 x1875450488705408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573904.069742:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488705920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573904.069747:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488705408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573904.069774:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573904.069779:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573904.069784:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488705408:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573904.069814:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488705536:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573904.069837:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573904.069866:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573904/real 1788573904] req@ffff94d4049bfb80 x1875450488705536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573904.069881:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488705536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573904.069890:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573904.069893:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573904.069897:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488705536:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573904.069903:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488705664:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573904.069922:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573904.069933:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573904/real 1788573904] req@ffff94d4049bea00 x1875450488705664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573904.069945:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488705664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573904.069960:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573904.069963:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573904.069967:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488705664:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573904.069973:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488705792:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573904.069992:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573904.070003:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573904/real 1788573904] req@ffff94d4049bf480 x1875450488705792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573904.070014:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488705792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573904.070023:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573904.070026:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573904.070029:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488705792:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573904.070035:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488705920:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573904.070052:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573904.070062:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573904/real 1788573904] req@ffff94d4049bc380 x1875450488705920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573904.070074:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc380 x1875450488705920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573904.070082:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573904.070085:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573904.070089:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488705920:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573904.581317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573905.029424:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573905.605308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573906.053441:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573906.629274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573907.077376:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573907.653340:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573908.101398:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573908.677321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573909.125413:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573909.189381:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573909.189392:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573909.189397:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573909.189409:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5057 00000100:00080000:1.0:1788573909.189423:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573909.189431:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573909.189476:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488706048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573909.189509:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573909.189512:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573909.189513:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573909.189516:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5057 00000100:00080000:1.0:1788573909.189519:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573909.189521:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573909.189530:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573909.189539:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488706176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573909.189545:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573909.189548:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573909.189549:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573909.189551:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5057 00000100:00080000:1.0:1788573909.189553:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573909.189555:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573909.189557:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573909.189566:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488706304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573909.189571:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573909.189573:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573909.189575:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573909.189577:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5057 00000100:00080000:1.0:1788573909.189581:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573909.189584:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573909.189588:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573909.189595:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488706048:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573909.189597:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488706432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573909.189609:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573909.189611:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573909.189613:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573909.189615:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5057 00000100:00080000:1.0:1788573909.189618:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573909.189620:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573909.189623:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573909.189630:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488706560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573909.189677:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573909.189716:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573909/real 1788573909] req@ffff94d4049bd180 x1875450488706048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573909.189732:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488706048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573909.189742:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573909.189745:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573909.189748:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488706048:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573909.189753:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488706176:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573909.189768:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573909.189776:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573909/real 1788573909] req@ffff94d4049bce00 x1875450488706176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573909.189784:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488706176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573909.189790:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573909.189793:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573909.189795:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488706176:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573909.189799:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488706304:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573909.189809:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573909.189815:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573909/real 1788573909] req@ffff94d4049bc380 x1875450488706304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573909.189836:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc380 x1875450488706304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573909.189845:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573909.189848:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573909.189853:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488706304:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573909.189883:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488706432:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573909.189902:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573909.189911:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573909/real 1788573909] req@ffff94d4049bf480 x1875450488706432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573909.189919:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488706432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573909.189925:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573909.189927:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573909.189929:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488706432:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573909.189932:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488706560:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573909.189943:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573909.189949:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573909/real 1788573909] req@ffff94d4049bea00 x1875450488706560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573909.189956:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488706560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573909.189961:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573909.189963:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573909.189965:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488706560:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573909.701310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573910.149379:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573910.725295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573911.173458:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573911.749326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573912.197398:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573912.773329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573913.221346:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573913.797315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573914.245342:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573914.309369:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573914.309377:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573914.309381:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573914.309389:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5062 00000100:00080000:2.0:1788573914.309399:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573914.309405:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573914.309437:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488706688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573914.309461:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573914.309464:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573914.309465:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573914.309468:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5062 00000100:00080000:2.0:1788573914.309470:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573914.309472:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573914.309479:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573914.309487:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488706816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573914.309494:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573914.309497:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573914.309499:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573914.309501:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5062 00000100:00080000:2.0:1788573914.309503:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573914.309506:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573914.309509:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573914.309517:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488706944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573914.309524:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573914.309526:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573914.309528:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573914.309531:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5062 00000100:00080000:2.0:1788573914.309533:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573914.309536:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573914.309539:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573914.309551:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488707072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573914.309553:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488706688:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788573914.309562:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573914.309572:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573914.309576:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573914.309581:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5062 00000100:00080000:2.0:1788573914.309584:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573914.309588:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573914.309593:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573914.309608:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488707200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573914.309675:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573914.309730:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573914/real 1788573914] req@ffff94d408c62d80 x1875450488706688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573914.309757:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488706688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573914.309770:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573914.309776:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573914.309781:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488706688:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573914.309794:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488706816:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573914.309823:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573914.309839:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573914/real 1788573914] req@ffff94d408c60000 x1875450488706816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573914.309854:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488706816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573914.309863:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573914.309867:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573914.309871:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488706816:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573914.309878:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488706944:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573914.309901:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573914.309915:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573914/real 1788573914] req@ffff94d408c63800 x1875450488706944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573914.309927:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488706944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573914.309937:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573914.309941:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573914.309945:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488706944:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573914.309981:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488707072:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573914.310004:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573914.310017:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573914/real 1788573914] req@ffff94d408c62680 x1875450488707072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573914.310031:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488707072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573914.310040:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573914.310043:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573914.310047:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488707072:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573914.310054:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488707200:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573914.310104:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573914.310120:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573914/real 1788573914] req@ffff94d408c61180 x1875450488707200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573914.310135:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488707200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573914.310144:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573914.310149:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573914.310152:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488707200:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573914.821302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573915.269360:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573915.845351:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573916.293399:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573916.869292:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573917.317330:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573917.893315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573918.341415:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573918.917265:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573919.365354:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573919.429327:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573919.429338:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573919.429343:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573919.429354:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5067 00000100:00080000:1.0:1788573919.429365:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573919.429374:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573919.429407:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488707328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573919.429431:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573919.429434:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573919.429436:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573919.429438:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5067 00000100:00080000:1.0:1788573919.429441:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573919.429443:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573919.429450:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573919.429459:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488707456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573919.429465:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573919.429469:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573919.429470:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573919.429472:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5067 00000100:00080000:1.0:1788573919.429474:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573919.429479:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573919.429480:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488707328:192.168.202.125@tcp:250:kworker.0 00000008:00100000:1.0:1788573919.429482:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573919.429492:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488707584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573919.429498:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573919.429500:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573919.429502:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573919.429504:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5067 00000100:00080000:1.0:1788573919.429507:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573919.429510:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573919.429513:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573919.429522:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488707712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573919.429528:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573919.429530:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573919.429531:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573919.429534:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5067 00000100:00080000:1.0:1788573919.429536:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573919.429539:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573919.429542:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573919.429553:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488707840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573919.429557:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573919.429598:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573919/real 1788573919] req@ffff94d4049bfb80 x1875450488707328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573919.429612:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488707328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573919.429620:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573919.429623:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573919.429626:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488707328:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573919.429631:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488707456:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573919.429647:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573919.429654:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573919/real 1788573919] req@ffff94d4049bc000 x1875450488707456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573919.429662:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488707456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573919.429667:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573919.429670:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573919.429672:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488707456:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573919.429689:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488707584:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573919.429700:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573919.429706:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573919/real 1788573919] req@ffff94d4049bc700 x1875450488707584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573919.429718:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488707584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573919.429723:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573919.429725:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573919.429727:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488707584:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573919.429731:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488707712:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573919.429741:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573919.429746:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573919/real 1788573919] req@ffff94d4049bed80 x1875450488707712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573919.429753:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488707712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573919.429758:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573919.429760:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573919.429762:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488707712:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573919.429765:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488707840:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573919.429774:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573919.429779:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573919/real 1788573919] req@ffff94d4049bf800 x1875450488707840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573919.429786:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488707840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573919.429795:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573919.429797:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573919.429799:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488707840:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573919.941394:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573920.389574:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573920.965301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573921.413368:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573921.989321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573922.437375:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573923.013362:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573923.461409:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573924.037320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573924.485453:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573924.549345:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573924.549356:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573924.549360:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573924.549371:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5072 00000100:00080000:1.0:1788573924.549382:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573924.549391:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573924.549427:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488707968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573924.549458:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573924.549462:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573924.549465:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573924.549470:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5072 00000100:00080000:1.0:1788573924.549473:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573924.549478:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573924.549488:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573924.549489:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488707968:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573924.549509:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488708096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573924.549523:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573924.549528:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573924.549531:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573924.549536:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5072 00000100:00080000:1.0:1788573924.549543:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573924.549549:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573924.549555:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788573924.549561:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573924.549573:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488708224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573924.549586:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573924.549591:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573924.549593:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573924.549597:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5072 00000100:00100000:0.0:1788573924.549602:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573924/real 1788573924] req@ffff94d4049be680 x1875450488707968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573924.549604:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573924.549612:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573924.549622:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573924.549632:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488707968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573924.549638:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488708352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573924.549656:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573924.549658:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573924.549665:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573924.549666:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:1.0:1788573924.549668:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573924.549676:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488707968:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788573924.549677:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5072 00000100:00080000:1.0:1788573924.549684:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573924.549690:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573924.549695:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573924.549707:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488708480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573924.549710:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488708096:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573924.549756:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573924.549773:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573924/real 1788573924] req@ffff94d4049bf100 x1875450488708096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573924.549789:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488708096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573924.549800:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573924.549804:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573924.549808:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488708096:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573924.549815:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488708224:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573924.549841:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573924.549858:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573924/real 1788573924] req@ffff94d4049bca80 x1875450488708224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573924.549874:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488708224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573924.549886:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573924.549891:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573924.549897:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488708224:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573924.549906:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488708352:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573924.549928:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573924.549939:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573924/real 1788573924] req@ffff94d4049bd500 x1875450488708352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573924.549952:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488708352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573924.549961:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573924.549964:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573924.549968:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488708352:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573924.549983:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488708480:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573924.550001:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573924.550013:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573924/real 1788573924] req@ffff94d4049bdc00 x1875450488708480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573924.550025:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488708480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573924.550034:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573924.550038:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573924.550041:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488708480:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573925.061335:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573925.509385:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573926.085300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573926.533398:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573927.109312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573927.557374:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573928.133369:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573928.581396:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573929.157274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573929.605433:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573929.669317:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573929.669328:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573929.669332:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573929.669343:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5077 00000100:00080000:2.0:1788573929.669355:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573929.669363:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573929.669407:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60700 x1875450488708608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573929.669437:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573929.669441:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573929.669442:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573929.669445:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5077 00000100:00080000:2.0:1788573929.669447:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573929.669450:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573929.669459:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573929.669467:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60380 x1875450488708736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573929.669476:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573929.669479:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573929.669480:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573929.669483:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5077 00000100:00080000:2.0:1788573929.669486:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573929.669489:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573929.669492:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573929.669499:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62300 x1875450488708864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573929.669504:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488708608:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788573929.669508:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573929.669513:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573929.669514:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573929.669517:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5077 00000100:00080000:2.0:1788573929.669520:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573929.669523:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573929.669528:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573929.669539:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488708992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573929.669545:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573929.669548:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573929.669549:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573929.669552:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5077 00000100:00080000:2.0:1788573929.669555:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573929.669557:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573929.669561:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573929.669567:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488709120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573929.669584:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573929.669623:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573929/real 1788573929] req@ffff94d408c60700 x1875450488708608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573929.669638:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60700 x1875450488708608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573929.669647:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573929.669650:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573929.669654:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488708608:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573929.669660:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488708736:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573929.669675:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573929.669683:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573929/real 1788573929] req@ffff94d408c60380 x1875450488708736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573929.669692:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60380 x1875450488708736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573929.669697:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573929.669699:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573929.669701:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488708736:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573929.669733:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488708864:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573929.669746:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573929.669753:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573929/real 1788573929] req@ffff94d408c62300 x1875450488708864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573929.669761:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62300 x1875450488708864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573929.669772:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573929.669774:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573929.669776:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488708864:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573929.669779:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488708992:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573929.669790:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573929.669803:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573929/real 1788573929] req@ffff94d408c61180 x1875450488708992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573929.669811:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488708992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573929.669816:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573929.669818:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573929.669821:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488708992:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573929.669824:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488709120:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573929.669835:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573929.669840:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573929/real 1788573929] req@ffff94d408c62680 x1875450488709120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573929.669847:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488709120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573929.669852:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573929.669854:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573929.669857:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488709120:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573930.181369:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573930.629412:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573931.205268:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573931.653462:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573932.229328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573932.677451:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573933.253276:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573933.701378:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573934.277286:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573934.725425:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573934.789374:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573934.789400:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573934.789406:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573934.789418:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5083 00000100:00080000:2.0:1788573934.789431:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573934.789440:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573934.789500:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488709248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573934.789531:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573934.789537:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573934.789538:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573934.789541:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5083 00000100:00080000:2.0:1788573934.789544:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573934.789548:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573934.789557:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573934.789569:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488709376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573934.789577:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573934.789587:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573934.789589:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573934.789593:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5083 00000100:00080000:2.0:1788573934.789596:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573934.789598:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573934.789605:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573934.789616:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488709504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573934.789623:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573934.789628:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573934.789632:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573934.789635:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488709248:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788573934.789636:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5083 00000100:00080000:2.0:1788573934.789641:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573934.789648:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573934.789657:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573934.789668:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488709632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573934.789674:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573934.789678:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573934.789679:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573934.789683:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5083 00000100:00080000:2.0:1788573934.789684:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573934.789687:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573934.789690:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573934.789697:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488709760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573934.789736:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573934.789781:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573934/real 1788573934] req@ffff94d408c63800 x1875450488709248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573934.789801:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488709248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573934.789813:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573934.789817:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573934.789821:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488709248:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573934.789830:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488709376:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573934.789856:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573934.789869:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573934/real 1788573934] req@ffff94d408c60000 x1875450488709376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573934.789883:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488709376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573934.789892:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573934.789896:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573934.789899:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488709376:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573934.789939:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488709504:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788573934.789962:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573934.789975:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573934/real 1788573934] req@ffff94d408c62d80 x1875450488709504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573934.789988:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488709504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573934.789997:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573934.790000:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573934.790004:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488709504:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788573934.790010:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488709632:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573934.790031:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573934.790042:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573934/real 1788573934] req@ffff94d408c62a00 x1875450488709632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573934.790054:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488709632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573934.790062:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573934.790065:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573934.790068:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488709632:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788573934.790074:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488709760:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788573934.790093:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788573934.790106:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573934/real 1788573934] req@ffff94d408c60a80 x1875450488709760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573934.790123:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488709760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573934.790132:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573934.790135:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788573934.790138:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488709760:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573935.301291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573935.749441:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573936.325360:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573936.773369:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573937.349302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573937.797488:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573938.373295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573938.821353:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573939.397356:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573939.845479:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573939.909362:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573939.909381:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573939.909386:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573939.909395:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5088 00000100:00080000:0.0:1788573939.909405:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573939.909411:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573939.909445:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488709888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573939.909473:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573939.909476:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573939.909477:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573939.909480:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5088 00000100:00080000:0.0:1788573939.909482:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573939.909485:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573939.909492:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573939.909504:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488710016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573939.909513:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573939.909517:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573939.909518:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573939.909524:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5088 00000100:00100000:3.0:1788573939.909534:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488709888:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788573939.909535:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573939.909544:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573939.909550:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573939.909557:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488710144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573939.909565:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573939.909567:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573939.909568:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573939.909571:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5088 00000100:00080000:0.0:1788573939.909575:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573939.909578:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573939.909581:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573939.909589:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488710272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573939.909596:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573939.909599:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573939.909600:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573939.909602:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5088 00000100:00080000:0.0:1788573939.909607:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573939.909610:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573939.909613:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573939.909620:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488710400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573939.909630:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573939.909683:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573939/real 1788573939] req@ffff94d408d3ea00 x1875450488709888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573939.909701:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488709888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573939.909711:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573939.909715:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573939.909718:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488709888:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573939.909744:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488710016:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573939.909762:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573939.909770:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573939/real 1788573939] req@ffff94d408d3ce00 x1875450488710016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573939.909779:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488710016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573939.909785:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573939.909788:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573939.909790:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488710016:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573939.909794:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488710144:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573939.909805:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573939.909810:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573939/real 1788573939] req@ffff94d408d3df80 x1875450488710144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573939.909819:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488710144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573939.909825:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573939.909826:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573939.909828:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488710144:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573939.909832:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488710272:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573939.909844:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573939.909849:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573939/real 1788573939] req@ffff94d408d3d180 x1875450488710272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573939.909857:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488710272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573939.909862:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573939.909864:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573939.909866:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488710272:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573939.909870:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488710400:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573939.909879:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573939.909885:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573939/real 1788573939] req@ffff94d408d3c000 x1875450488710400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573939.909891:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488710400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573939.909901:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573939.909903:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573939.909906:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488710400:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573940.421307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573940.869415:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573941.445373:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573941.893422:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573942.469341:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573942.917433:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573943.493352:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573943.941449:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573944.517339:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573944.965414:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573945.029343:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573945.029354:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573945.029358:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573945.029370:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5093 00000100:00080000:0.0:1788573945.029383:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573945.029391:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573945.029459:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488710528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573945.029488:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573945.029492:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573945.029493:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573945.029496:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5093 00000100:00080000:0.0:1788573945.029498:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573945.029501:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573945.029510:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573945.029518:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488710656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573945.029532:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573945.029535:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573945.029536:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573945.029538:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5093 00000100:00080000:0.0:1788573945.029540:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573945.029543:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573945.029545:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573945.029551:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488710784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573945.029561:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788573945.029561:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488710528:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788573945.029566:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573945.029569:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573945.029575:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5093 00000100:00080000:0.0:1788573945.029577:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573945.029579:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573945.029583:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573945.029590:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488710912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573945.029595:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573945.029597:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573945.029599:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573945.029601:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5093 00000100:00080000:0.0:1788573945.029604:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573945.029607:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573945.029610:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573945.029617:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488711040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573945.029656:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573945.029705:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573945/real 1788573945] req@ffff94d408d3dc00 x1875450488710528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573945.029738:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488710528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573945.029751:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573945.029757:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573945.029762:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488710528:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573945.029771:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488710656:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573945.029799:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573945.029814:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573945/real 1788573945] req@ffff94d408d3c380 x1875450488710656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573945.029836:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488710656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573945.029845:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573945.029849:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573945.029853:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488710656:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573945.029883:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488710784:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573945.029906:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573945.029921:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573945/real 1788573945] req@ffff94d408d3f480 x1875450488710784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573945.029934:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488710784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573945.029943:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573945.029947:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573945.029951:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488710784:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573945.029958:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488710912:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573945.029981:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573945.029994:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573945/real 1788573945] req@ffff94d408d3c700 x1875450488710912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573945.030017:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488710912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573945.030032:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573945.030035:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573945.030039:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488710912:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573945.030046:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488711040:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573945.030069:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573945.030081:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573945/real 1788573945] req@ffff94d408d3e300 x1875450488711040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573945.030095:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488711040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573945.030103:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573945.030107:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573945.030111:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488711040:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573945.541341:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573945.989439:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573946.565344:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573947.013432:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573947.589371:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573948.037452:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573948.613307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573949.061363:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573949.637311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573950.085375:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573950.149379:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573950.149391:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573950.149396:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573950.149409:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5098 00000100:00080000:2.0:1788573950.149429:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573950.149437:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573950.149503:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488711168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573950.149535:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573950.149540:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573950.149542:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573950.149546:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5098 00000100:00080000:2.0:1788573950.149551:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573950.149556:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573950.149569:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573950.149585:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488711296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573950.149598:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573950.149603:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573950.149605:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573950.149610:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5098 00000100:00080000:2.0:1788573950.149614:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573950.149619:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573950.149624:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488711168:192.168.202.125@tcp:250:kworker.0 00000008:00100000:2.0:1788573950.149627:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573950.149644:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488711424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573950.149657:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573950.149661:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573950.149663:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573950.149668:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5098 00000100:00080000:2.0:1788573950.149691:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573950.149698:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573950.149704:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573950.149718:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488711552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573950.149729:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488711296:192.168.202.125@tcp:38:kworker.0 00000100:00080000:2.0:1788573950.149746:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573950.149750:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573950.149752:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573950.149757:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5098 00000100:00080000:2.0:1788573950.149761:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573950.149765:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573950.149770:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573950.149772:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488711424:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788573950.149786:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488711680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573950.149797:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488711552:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573950.149830:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488711680:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573950.661269:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573951.109395:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573951.685335:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573952.133340:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573952.709271:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573953.157381:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788573953.221461:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:1.0:1788573953.221503:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788573953.221517:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788573953.221571:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788573953.221583:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788573953.221594:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788573953.221606:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:3.0:1788573953.221627:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573950/real 1788573953] req@ffff94d408c63480 x1875450488711424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573953.221654:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488711424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573953.221668:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573953.221675:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573953.221680:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488711424:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573953.221690:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573950/real 1788573953] req@ffff94d408c61f80 x1875450488711552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573953.221708:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488711552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573953.221717:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573953.221721:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573953.221726:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488711552:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573953.221734:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573950/real 1788573953] req@ffff94d408c63b80 x1875450488711168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573953.221758:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488711168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573953.221766:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573953.221770:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573953.221775:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488711168:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573953.221790:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573950/real 1788573953] req@ffff94d408c61500 x1875450488711296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573953.221804:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488711296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573953.221811:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573953.221814:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573953.221818:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488711296:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573953.221847:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573950/real 1788573953] req@ffff94d408c61c00 x1875450488711680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573953.221861:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488711680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573953.221871:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573953.221875:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573953.221878:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488711680:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573953.733274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573954.181363:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573954.757261:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573955.205408:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573955.269354:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573955.269373:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573955.269379:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573955.269392:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5103 00000100:00080000:1.0:1788573955.269406:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573955.269414:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573955.269456:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd880 x1875450488711808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573955.269499:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573955.269503:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573955.269505:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573955.269510:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5103 00000100:00080000:1.0:1788573955.269514:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573955.269519:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573955.269530:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573955.269541:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488711808:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788573955.269561:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488711936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573955.269574:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573955.269579:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573955.269581:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573955.269585:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5103 00000100:00080000:1.0:1788573955.269589:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573955.269593:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573955.269598:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573955.269613:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488712064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573955.269624:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573955.269628:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573955.269629:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573955.269633:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5103 00000100:00080000:1.0:1788573955.269637:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573955.269642:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:3.0:1788573955.269649:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:1.0:1788573955.269652:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573955.269669:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488712192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573955.269679:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573955.269683:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573955.269685:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573955.269691:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5103 00000100:00080000:1.0:1788573955.269696:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788573955.269702:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573955/real 1788573955] req@ffff94d4049bd880 x1875450488711808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573955.269704:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573955.269716:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573955.269731:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488712320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573955.269735:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd880 x1875450488711808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573955.269760:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573955.269766:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573955.269771:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488711808:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573955.269803:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488711936:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573955.269828:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573955.269844:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573955/real 1788573955] req@ffff94d4049be300 x1875450488711936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573955.269857:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488711936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573955.269864:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573955.269867:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573955.269871:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488711936:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573955.269877:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488712064:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573955.269898:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573955.269911:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573955/real 1788573955] req@ffff94d4049bdc00 x1875450488712064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573955.269923:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488712064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573955.269932:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573955.269935:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573955.269939:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488712064:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573955.269946:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488712192:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573955.269968:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573955.269980:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573955/real 1788573955] req@ffff94d4049bd500 x1875450488712192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573955.269993:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488712192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573955.270000:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573955.270085:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573955.270090:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488712192:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573955.270097:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488712320:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573955.270121:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573955.270135:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573955/real 1788573955] req@ffff94d4049bca80 x1875450488712320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573955.270148:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488712320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573955.270157:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573955.270160:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573955.270164:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488712320:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573955.781314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573956.229334:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573956.805355:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573957.253350:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573957.829288:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573958.277328:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573958.853422:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573959.301415:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573959.877830:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788573960.325445:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573960.389355:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573960.389363:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573960.389367:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573960.389378:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5108 00000100:00080000:1.0:1788573960.389388:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573960.389394:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788573960.389430:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488712448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573960.389469:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573960.389472:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573960.389474:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573960.389478:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5108 00000100:00080000:1.0:1788573960.389482:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573960.389488:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573960.389496:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573960.389508:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488712576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573960.389521:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788573960.389521:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488712448:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788573960.389528:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573960.389530:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573960.389539:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5108 00000100:00080000:1.0:1788573960.389542:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573960.389544:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573960.389547:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573960.389556:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488712704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573960.389563:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573960.389567:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573960.389569:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573960.389573:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5108 00000100:00080000:1.0:1788573960.389576:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573960.389578:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573960.389582:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573960.389592:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488712832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573960.389600:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573960.389602:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573960.389604:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573960.389628:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5108 00000100:00080000:1.0:1788573960.389632:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573960.389635:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788573960.389639:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788573960.389646:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788573960.389653:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488712960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573960.389704:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573960/real 1788573960] req@ffff94d4049bf100 x1875450488712448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573960.389731:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488712448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573960.389742:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573960.389755:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573960.389763:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488712448:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788573960.389792:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488712576:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788573960.389813:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573960.389823:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573960/real 1788573960] req@ffff94d4049be680 x1875450488712576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573960.389834:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488712576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573960.389842:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573960.389845:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573960.389847:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488712576:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788573960.389852:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488712704:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788573960.389872:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573960.389889:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573960/real 1788573960] req@ffff94d4049bf800 x1875450488712704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573960.389898:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488712704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573960.389912:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573960.389915:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573960.389918:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488712704:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788573960.389923:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488712832:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788573960.389957:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573960.389977:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573960/real 1788573960] req@ffff94d4049bed80 x1875450488712832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573960.389986:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488712832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573960.389996:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573960.389998:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573960.390000:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488712832:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788573960.390004:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488712960:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788573960.390016:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573960.390022:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573960/real 1788573960] req@ffff94d4049bc700 x1875450488712960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573960.390029:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488712960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573960.390035:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573960.390037:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788573960.390038:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488712960:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573960.901465:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573961.349456:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573961.925534:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573962.373401:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573962.949779:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573963.397380:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573963.973295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573964.421367:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573964.997315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573965.445377:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573965.509324:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573965.509334:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573965.509339:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573965.509350:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5113 00000100:00080000:2.0:1788573965.509361:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573965.509370:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573965.509408:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60e00 x1875450488713088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573965.509435:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573965.509440:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573965.509443:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573965.509448:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5113 00000100:00080000:2.0:1788573965.509452:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573965.509457:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573965.509467:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573965.509469:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488713088:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788573965.509485:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63100 x1875450488713216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573965.509499:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573965.509503:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573965.509505:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573965.509509:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5113 00000100:00080000:2.0:1788573965.509513:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573965.509517:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573965.509523:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573965.509535:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488713344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573965.509545:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788573965.509549:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788573965.509551:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573965.509555:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573965.509560:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5113 00000100:00080000:2.0:1788573965.509564:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573965.509568:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573965.509574:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573965.509585:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488713472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573965.509592:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573965.509594:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573965.509598:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573965.509599:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573965/real 1788573965] req@ffff94d408c60e00 x1875450488713088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573965.509602:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5113 00000100:00080000:2.0:1788573965.509606:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573965.509619:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573965.509623:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573965.509627:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60e00 x1875450488713088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573965.509633:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488713600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573965.509656:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573965.509664:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573965.509682:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488713088:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573965.509710:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488713216:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573965.509736:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573965.509753:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573965/real 1788573965] req@ffff94d408c63100 x1875450488713216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573965.509779:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63100 x1875450488713216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573965.509789:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573965.509793:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573965.509797:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488713216:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573965.509804:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488713344:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573965.509823:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573965.509835:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573965/real 1788573965] req@ffff94d408c61880 x1875450488713344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573965.509848:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488713344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573965.509858:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573965.509862:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573965.509866:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488713344:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573965.509873:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488713472:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573965.509893:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573965.509904:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573965/real 1788573965] req@ffff94d408c61c00 x1875450488713472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573965.509931:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488713472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573965.509940:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573965.509944:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573965.509947:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488713472:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573965.509954:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488713600:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573965.509972:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573965.509982:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573965/real 1788573965] req@ffff94d408c61500 x1875450488713600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573965.509994:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488713600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573965.510003:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573965.510006:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573965.510010:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488713600:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573966.021280:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573966.469370:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573967.045321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573967.493418:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573968.069290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573968.517348:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573969.093302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573969.541379:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573970.117289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573970.565364:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573970.629297:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573970.629317:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573970.629322:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573970.629333:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5118 00000100:00080000:0.0:1788573970.629345:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573970.629354:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573970.629398:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ed80 x1875450488713728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573970.629435:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573970.629440:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573970.629443:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573970.629447:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5118 00000100:00080000:0.0:1788573970.629451:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573970.629456:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573970.629468:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573970.629483:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488713856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573970.629495:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573970.629499:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573970.629502:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573970.629508:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5118 00000100:00080000:0.0:1788573970.629512:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573970.629517:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573970.629526:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573970.629527:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488713728:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788573970.629547:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488713984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573970.629563:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573970.629567:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573970.629570:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573970.629576:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5118 00000100:00080000:0.0:1788573970.629581:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573970.629586:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573970.629591:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573970.629603:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488714112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573970.629614:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788573970.629615:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788573970.629630:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573970.629632:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573970.629637:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5118 00000100:00080000:0.0:1788573970.629640:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573970.629644:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573970.629648:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573970.629659:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488714240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573970.629666:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573970/real 1788573970] req@ffff94d408d3ed80 x1875450488713728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573970.629696:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ed80 x1875450488713728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573970.629706:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573970.629709:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573970.629714:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488713728:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573970.629723:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488713856:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573970.629780:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573970.629795:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573970/real 1788573970] req@ffff94d408d3d880 x1875450488713856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573970.629809:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488713856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573970.629819:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573970.629823:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573970.629827:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488713856:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573970.629853:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488713984:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573970.629875:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573970.629887:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573970/real 1788573970] req@ffff94d408d3e300 x1875450488713984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573970.629909:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488713984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573970.629919:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573970.629923:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573970.629927:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488713984:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573970.629934:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488714112:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573970.629955:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573970.629966:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573970/real 1788573970] req@ffff94d408d3c700 x1875450488714112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573970.629980:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488714112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573970.629989:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573970.629992:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573970.629996:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488714112:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573970.630003:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488714240:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573970.630021:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573970.630032:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573970/real 1788573970] req@ffff94d408d3f480 x1875450488714240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573970.630044:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488714240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573970.630059:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573970.630063:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573970.630067:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488714240:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573971.141275:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573971.589384:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573972.165282:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573972.613364:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573973.189299:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573973.637375:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573974.213386:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573974.661446:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573975.237298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573975.685340:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573975.749346:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573975.749357:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573975.749361:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573975.749371:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5124 00000100:00080000:2.0:1788573975.749383:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573975.749391:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573975.749439:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488714368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573975.749466:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573975.749471:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573975.749472:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573975.749476:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5124 00000100:00080000:2.0:1788573975.749481:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573975.749485:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573975.749495:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573975.749517:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488714496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573975.749520:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488714368:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788573975.749538:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573975.749542:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573975.749545:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573975.749550:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5124 00000100:00080000:2.0:1788573975.749554:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573975.749558:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573975.749564:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573975.749576:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488714624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573975.749586:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788573975.749590:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788573975.749592:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573975.749596:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573975.749601:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5124 00000100:00080000:2.0:1788573975.749606:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573975.749618:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573975.749624:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573975.749633:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573975/real 1788573975] req@ffff94d408c63b80 x1875450488714368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573975.749639:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488714752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573975.749657:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488714368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573975.749666:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573975.749671:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573975.749672:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573975.749678:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573975.749678:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573975.749686:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488714368:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788573975.749687:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5124 00000100:00080000:2.0:1788573975.749695:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573975.749700:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573975.749706:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573975.749718:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488714496:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788573975.749721:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488714880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573975.749742:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573975.749753:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573975/real 1788573975] req@ffff94d408c61f80 x1875450488714496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573975.749768:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488714496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573975.749775:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573975.749792:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573975.749795:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488714496:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573975.749800:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488714624:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573975.749815:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573975.749836:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573975/real 1788573975] req@ffff94d408c63480 x1875450488714624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573975.749851:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488714624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573975.749860:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573975.749864:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573975.749868:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488714624:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573975.749875:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488714752:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573975.749895:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573975.749903:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573975/real 1788573975] req@ffff94d408c60a80 x1875450488714752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573975.749911:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488714752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573975.749921:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573975.749924:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573975.749926:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488714752:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573975.749939:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488714880:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573975.749957:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573975.749964:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573975/real 1788573975] req@ffff94d408c62a00 x1875450488714880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573975.749972:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488714880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573975.749978:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573975.749980:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573975.749982:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488714880:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573976.261327:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573976.709368:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573977.285284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573977.733333:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573978.309409:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573978.757378:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573979.333338:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573979.781408:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573980.357351:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573980.805403:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573980.869366:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573980.869377:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573980.869381:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573980.869391:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5129 00000100:00080000:2.0:1788573980.869400:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573980.869408:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573980.869449:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488715008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573980.869478:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573980.869483:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573980.869485:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573980.869489:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5129 00000100:00080000:2.0:1788573980.869493:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573980.869498:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573980.869508:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573980.869526:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488715136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573980.869535:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488715008:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788573980.869542:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573980.869547:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573980.869550:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573980.869555:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5129 00000100:00080000:2.0:1788573980.869559:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573980.869563:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573980.869569:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573980.869584:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488715264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573980.869595:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573980.869598:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573980.869600:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788573980.869605:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788573980.869610:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5129 00000100:00080000:2.0:1788573980.869616:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573980.869621:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573980.869629:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573980.869640:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573980/real 1788573980] req@ffff94d408c62d80 x1875450488715008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573980.869644:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488715392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573980.869680:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573980.869684:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1788573980.869687:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488715008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573980.869689:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573980.869699:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5129 00000100:00080000:3.0:1788573980.869703:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573980.869705:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573980.869708:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:2.0:1788573980.869716:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573980.869717:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488715008:192.168.202.125@tcp:250:kworker.0 00000008:00100000:2.0:1788573980.869728:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573980.869741:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488715136:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788573980.869759:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488715520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573980.869764:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573980.869775:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573980/real 1788573980] req@ffff94d408c60000 x1875450488715136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573980.869786:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488715136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573980.869793:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573980.869815:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573980.869818:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488715136:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573980.869823:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488715264:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573980.869836:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573980.869843:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573980/real 1788573980] req@ffff94d408c63800 x1875450488715264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573980.869852:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488715264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573980.869862:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573980.869864:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573980.869866:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488715264:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573980.869870:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488715392:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573980.869880:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573980.869886:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573980/real 1788573980] req@ffff94d408c62680 x1875450488715392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573980.869893:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488715392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573980.869899:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573980.869900:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573980.869902:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488715392:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573980.869912:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488715520:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573980.869922:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573980.869928:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573980/real 1788573980] req@ffff94d408c61180 x1875450488715520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573980.869936:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488715520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573980.869941:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573980.869942:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573980.869945:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488715520:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573981.381272:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573981.829398:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573982.405352:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573982.853388:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573983.429309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573983.877350:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573984.453359:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573984.901380:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573985.477366:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573985.925362:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573985.989324:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573985.989332:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573985.989336:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573985.989345:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5134 00000100:00080000:0.0:1788573985.989354:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573985.989360:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788573985.989393:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488715648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573985.989417:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573985.989420:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573985.989421:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573985.989424:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5134 00000100:00080000:0.0:1788573985.989426:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573985.989429:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573985.989436:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573985.989444:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488715776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573985.989453:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573985.989455:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573985.989457:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573985.989462:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5134 00000100:00100000:3.0:1788573985.989463:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488715648:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788573985.989464:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573985.989470:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573985.989476:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573985.989484:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488715904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573985.989490:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573985.989493:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573985.989495:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573985.989498:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5134 00000100:00080000:0.0:1788573985.989500:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573985.989503:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573985.989506:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573985.989512:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488716032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573985.989518:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573985.989520:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573985.989521:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573985.989524:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5134 00000100:00080000:0.0:1788573985.989526:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788573985.989530:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788573985.989533:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788573985.989537:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573985.989544:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488716160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573985.989567:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573985/real 1788573985] req@ffff94d408d3c380 x1875450488715648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573985.989592:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488715648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573985.989600:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573985.989603:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573985.989606:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488715648:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573985.989627:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488715776:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573985.989643:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573985.989652:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573985/real 1788573985] req@ffff94d408d3dc00 x1875450488715776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573985.989660:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488715776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573985.989666:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573985.989669:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573985.989671:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488715776:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573985.989675:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488715904:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573985.989686:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573985.989691:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573985/real 1788573985] req@ffff94d408d3c000 x1875450488715904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573985.989699:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488715904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573985.989704:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573985.989706:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573985.989708:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488715904:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573985.989711:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488716032:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573985.989721:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573985.989726:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573985/real 1788573985] req@ffff94d408d3d180 x1875450488716032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573985.989734:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488716032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573985.989739:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573985.989741:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573985.989742:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488716032:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573985.989746:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488716160:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573985.989755:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573985.989760:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573985/real 1788573985] req@ffff94d408d3df80 x1875450488716160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573985.989767:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488716160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573985.989772:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573985.989775:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573985.989777:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488716160:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573986.501299:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573986.949392:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573987.525329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573987.973349:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573988.549334:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573988.997362:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573989.573326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573990.021365:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573990.597301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573991.045423:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573991.109417:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573991.109433:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573991.109438:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573991.109451:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5139 00000100:00080000:2.0:1788573991.109465:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573991.109473:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573991.109520:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62300 x1875450488716288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573991.109548:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573991.109552:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573991.109553:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573991.109556:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5139 00000100:00080000:2.0:1788573991.109558:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573991.109561:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573991.109569:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573991.109588:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60380 x1875450488716416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573991.109605:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573991.109607:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573991.109609:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573991.109614:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5139 00000100:00100000:3.0:1788573991.109616:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488716288:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788573991.109617:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573991.109623:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573991.109628:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573991.109637:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60700 x1875450488716544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573991.109644:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573991.109647:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573991.109648:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573991.109650:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5139 00000100:00080000:2.0:1788573991.109653:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573991.109655:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573991.109659:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573991.109668:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488716672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573991.109675:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573991.109677:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573991.109679:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573991.109682:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5139 00000100:00080000:2.0:1788573991.109685:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573991.109688:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573991.109692:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788573991.109696:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788573991.109704:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488716800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573991.109741:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573991/real 1788573991] req@ffff94d408c62300 x1875450488716288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573991.109760:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62300 x1875450488716288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573991.109772:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573991.109776:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573991.109781:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488716288:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573991.109813:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488716416:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573991.109848:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573991.109860:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573991/real 1788573991] req@ffff94d408c60380 x1875450488716416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573991.109869:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60380 x1875450488716416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573991.109875:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573991.109877:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573991.109880:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488716416:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573991.109883:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488716544:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573991.109897:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573991.109903:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573991/real 1788573991] req@ffff94d408c60700 x1875450488716544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573991.109910:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60700 x1875450488716544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573991.109916:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573991.109917:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573991.109919:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488716544:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573991.109923:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488716672:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573991.109935:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573991.109941:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573991/real 1788573991] req@ffff94d408c61180 x1875450488716672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573991.109948:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488716672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573991.109953:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573991.109955:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573991.109957:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488716672:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573991.109960:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488716800:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573991.109970:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573991.109976:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573991/real 1788573991] req@ffff94d408c62680 x1875450488716800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573991.109983:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488716800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573991.109988:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573991.109990:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573991.109991:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488716800:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573991.621306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573992.069404:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573992.645300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573993.093364:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573993.669283:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573994.117462:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573994.693314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573995.141318:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573995.717321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573996.165381:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573996.229282:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573996.229290:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573996.229294:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573996.229302:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5144 00000100:00080000:2.0:1788573996.229311:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573996.229317:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788573996.229356:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488716928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573996.229387:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573996.229391:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573996.229393:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573996.229397:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5144 00000100:00080000:2.0:1788573996.229401:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573996.229406:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573996.229417:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573996.229435:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488717056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573996.229435:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488716928:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788573996.229458:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573996.229464:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573996.229467:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573996.229471:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5144 00000100:00080000:2.0:1788573996.229476:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573996.229481:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573996.229487:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573996.229503:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488717184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573996.229513:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788573996.229518:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573996.229521:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573996.229523:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573996.229527:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5144 00000100:00080000:2.0:1788573996.229531:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573996.229535:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788573996.229539:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573996.229549:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488717312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573996.229557:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573996/real 1788573996] req@ffff94d408c63800 x1875450488716928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573996.229558:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573996.229566:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573996.229570:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573996.229577:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5144 00000100:00080000:2.0:1788573996.229581:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573996.229583:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788573996.229588:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488716928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788573996.229589:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573996.229601:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488717440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573996.229602:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573996.229611:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573996.229619:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488716928:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788573996.229642:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488717056:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573996.229661:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573996.229672:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573996/real 1788573996] req@ffff94d408c60000 x1875450488717056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573996.229683:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488717056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573996.229689:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573996.229692:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573996.229694:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488717056:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573996.229698:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488717184:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788573996.229714:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573996.229721:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573996/real 1788573996] req@ffff94d408c62d80 x1875450488717184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573996.229735:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488717184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573996.229741:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573996.229743:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573996.229745:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488717184:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788573996.229749:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488717312:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573996.229766:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573996.229778:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573996/real 1788573996] req@ffff94d408c62a00 x1875450488717312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573996.229794:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488717312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573996.229804:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573996.229808:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573996.229813:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488717312:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573996.229821:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488717440:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788573996.229845:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788573996.229876:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573996/real 1788573996] req@ffff94d408c60a80 x1875450488717440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573996.229890:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488717440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573996.229963:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573996.229967:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788573996.229970:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488717440:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788573996.741278:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573997.189327:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573997.765392:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788573998.213397:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573998.789278:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788573999.237331:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573999.813352:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574000.261338:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574000.837341:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574001.285422:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574001.349317:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574001.349325:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574001.349329:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574001.349337:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5149 00000100:00080000:3.0:1788574001.349346:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574001.349353:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574001.349386:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488717568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574001.349409:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574001.349412:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574001.349414:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574001.349417:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5149 00000100:00080000:3.0:1788574001.349419:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574001.349422:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574001.349431:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574001.349443:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488717696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574001.349444:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488717568:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574001.349458:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574001.349461:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574001.349462:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574001.349464:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5149 00000100:00080000:3.0:1788574001.349466:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574001.349468:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574001.349471:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574001.349481:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488717824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574001.349490:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574001.349493:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574001.349495:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574001.349499:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5149 00000100:00080000:3.0:1788574001.349503:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788574001.349507:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574001.349512:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574001.349520:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574001.349534:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488717952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574001.349539:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574001/real 1788574001] req@ffff94d408c56a00 x1875450488717568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574001.349559:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574001.349564:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574001.349567:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574001.349571:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488717568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574001.349575:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5149 00000100:00080000:3.0:1788574001.349582:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574001.349588:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00080000:2.0:1788574001.349592:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788574001.349593:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788574001.349601:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574001.349607:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488718080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574001.349609:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488717568:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574001.349643:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488717696:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574001.349669:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574001.349680:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574001/real 1788574001] req@ffff94d408c55180 x1875450488717696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574001.349689:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488717696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574001.349695:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574001.349698:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574001.349700:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488717696:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574001.349705:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488717824:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574001.349717:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574001.349723:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574001/real 1788574001] req@ffff94d408c57480 x1875450488717824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574001.349731:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488717824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574001.349736:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574001.349738:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574001.349740:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488717824:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574001.349743:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488717952:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574001.349754:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574001.349759:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574001/real 1788574001] req@ffff94d408c56300 x1875450488717952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574001.349766:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488717952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574001.349771:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574001.349773:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574001.349775:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488717952:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574001.349778:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488718080:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574001.349787:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574001.349792:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574001/real 1788574001] req@ffff94d408c54000 x1875450488718080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574001.349805:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488718080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574001.349814:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574001.349817:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574001.349820:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488718080:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574001.861300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574002.309420:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574002.885303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574003.333372:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574003.909320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574004.357381:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574004.933306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574005.381341:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574005.957324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574006.405416:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574006.469331:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574006.469342:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574006.469346:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574006.469357:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5154 00000100:00080000:3.0:1788574006.469370:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574006.469386:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574006.469432:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488718208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574006.469459:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574006.469462:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574006.469463:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574006.469466:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5154 00000100:00080000:3.0:1788574006.469468:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574006.469471:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574006.469481:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574006.469490:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488718336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574006.469497:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574006.469500:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574006.469502:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574006.469507:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5154 00000100:00080000:3.0:1788574006.469509:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574006.469511:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574006.469514:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574006.469522:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488718464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574006.469529:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574006.469532:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574006.469533:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574006.469536:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5154 00000100:00080000:3.0:1788574006.469539:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574006.469545:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574006.469549:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574006.469550:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488718208:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574006.469558:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488718592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574006.469568:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574006.469571:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574006.469572:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574006.469574:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5154 00000100:00080000:3.0:1788574006.469576:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574006.469579:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574006.469581:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574006.469588:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488718720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574006.469637:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574006.469679:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574006/real 1788574006] req@ffff94d408c55f80 x1875450488718208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574006.469700:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488718208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574006.469712:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574006.469717:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574006.469721:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488718208:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574006.469731:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488718336:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574006.469757:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574006.469771:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574006/real 1788574006] req@ffff94d408c54700 x1875450488718336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574006.469780:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488718336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574006.469790:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574006.469792:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574006.469795:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488718336:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574006.469817:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488718464:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574006.469842:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574006.469853:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574006/real 1788574006] req@ffff94d408c54000 x1875450488718464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574006.469865:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488718464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574006.469874:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574006.469878:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574006.469881:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488718464:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574006.469916:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488718592:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574006.469940:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574006.469953:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574006/real 1788574006] req@ffff94d408c56300 x1875450488718592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574006.469965:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488718592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574006.469971:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574006.469973:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574006.469976:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488718592:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574006.469981:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488718720:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574006.469996:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574006.470003:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574006/real 1788574006] req@ffff94d408c57480 x1875450488718720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574006.470011:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488718720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574006.470016:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574006.470018:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574006.470020:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488718720:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574006.981286:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574007.429365:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574008.005352:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574008.453358:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574009.029272:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574009.477364:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574010.053332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574010.501389:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574011.077288:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574011.525394:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574011.589326:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574011.589337:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574011.589341:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574011.589352:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5159 00000100:00080000:0.0:1788574011.589365:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574011.589373:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574011.589416:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488718848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574011.589445:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574011.589448:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574011.589450:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574011.589452:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5159 00000100:00080000:0.0:1788574011.589455:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574011.589457:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574011.589466:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574011.589478:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488718976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574011.589486:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574011.589489:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574011.589490:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574011.589494:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5159 00000100:00080000:0.0:1788574011.589496:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574011.589499:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574011.589502:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574011.589508:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488719104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574011.589517:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574011.589519:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574011.589521:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574011.589521:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488718848:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788574011.589539:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5159 00000100:00080000:0.0:1788574011.589542:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574011.589546:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574011.589550:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574011.589560:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488719232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574011.589568:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574011.589571:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574011.589573:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574011.589577:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5159 00000100:00080000:0.0:1788574011.589580:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574011.589583:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574011.589586:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574011.589593:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488719360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574011.589607:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574011.589647:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574011/real 1788574011] req@ffff94d408d3ce00 x1875450488718848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574011.589667:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488718848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574011.589680:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574011.589684:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574011.589688:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488718848:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574011.589695:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488718976:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574011.589716:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574011.589725:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574011/real 1788574011] req@ffff94d408d3ea00 x1875450488718976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574011.589734:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488718976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574011.589740:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574011.589742:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574011.589744:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488718976:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574011.589765:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488719104:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574011.589776:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574011.589783:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574011/real 1788574011] req@ffff94d408d3e680 x1875450488719104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574011.589790:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488719104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574011.589796:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574011.589798:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574011.589800:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488719104:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574011.589804:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488719232:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574011.589814:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574011.589832:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574011/real 1788574011] req@ffff94d408d3d500 x1875450488719232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574011.589840:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488719232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574011.589848:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574011.589851:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574011.589855:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488719232:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574011.589861:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488719360:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574011.589880:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574011.589892:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574011/real 1788574011] req@ffff94d408d3ca80 x1875450488719360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574011.589903:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488719360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574011.589930:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574011.589933:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574011.589935:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488719360:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574012.101284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574012.549338:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574013.125306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574013.573382:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574014.149316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574014.597329:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574015.173351:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574015.621404:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574016.197333:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574016.645349:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574016.709366:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574016.709378:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574016.709383:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574016.709397:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5165 00000100:00080000:3.0:1788574016.709410:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574016.709418:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574016.709482:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488719488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574016.709516:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574016.709520:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574016.709529:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574016.709534:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5165 00000100:00080000:3.0:1788574016.709540:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574016.709546:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574016.709558:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574016.709572:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488719616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574016.709582:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574016.709585:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574016.709588:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574016.709598:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488719488:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574016.709599:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5165 00000100:00080000:3.0:1788574016.709605:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574016.709612:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574016.709617:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574016.709628:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488719744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574016.709636:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574016.709638:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574016.709640:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574016.709643:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5165 00000100:00080000:3.0:1788574016.709646:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574016.709650:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574016.709654:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574016.709662:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488719872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574016.709676:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574016.709678:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574016.709680:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574016.709683:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5165 00000100:00080000:3.0:1788574016.709686:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574016.709688:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574016.709691:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574016.709699:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488720000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574016.709746:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488719616:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574016.709787:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488719744:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574016.709814:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488719872:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574016.709836:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488720000:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574017.221364:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574017.669418:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574018.245282:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574018.693326:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574019.269273:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574019.717374:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788574019.781483:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:0.0:1788574019.781524:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788574019.781537:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:0.0:1788574019.781574:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:0.0:1788574019.781584:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:0.0:1788574019.781595:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:0.0:1788574019.781612:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:2.0:1788574019.781629:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574016/real 1788574019] req@ffff94d408c56a00 x1875450488719616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574019.781677:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488719616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574019.781690:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574019.781695:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574019.781700:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488719616:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574019.781709:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574016/real 1788574019] req@ffff94d408c55880 x1875450488719744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574019.781720:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488719744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574019.781728:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574019.781732:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574019.781735:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488719744:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574019.781742:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574016/real 1788574019] req@ffff94d408c55c00 x1875450488719872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574019.781753:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488719872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574019.781760:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574019.781763:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574019.781767:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488719872:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574019.781774:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574016/real 1788574019] req@ffff94d408c57800 x1875450488720000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574019.781784:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488720000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574019.781791:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574019.781794:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574019.781798:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488720000:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574019.781804:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574016/real 1788574019] req@ffff94d408c55180 x1875450488719488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574019.781816:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488719488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574019.781824:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574019.781827:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574019.781830:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488719488:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574020.293268:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574020.741352:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574021.317265:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574021.765381:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574021.829278:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574021.829290:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574021.829294:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574021.829305:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5170 00000100:00080000:3.0:1788574021.829319:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574021.829327:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574021.829367:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488720128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574021.829396:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574021.829399:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574021.829401:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574021.829404:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5170 00000100:00080000:3.0:1788574021.829406:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574021.829408:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574021.829428:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574021.829452:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488720256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574021.829464:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574021.829468:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574021.829470:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574021.829475:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5170 00000100:00080000:3.0:1788574021.829480:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574021.829484:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574021.829490:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574021.829499:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488720128:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574021.829510:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488720384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574021.829525:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574021.829529:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574021.829532:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574021.829538:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5170 00000100:00080000:3.0:1788574021.829543:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574021.829548:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574021.829555:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574021.829571:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488720512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574021.829582:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574021.829586:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574021.829589:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788574021.829594:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574021.829601:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5170 00000100:00080000:3.0:1788574021.829608:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574021.829613:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574021.829620:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574021.829636:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488720640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574021.829654:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574021/real 1788574021] req@ffff94d408c57100 x1875450488720128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574021.829681:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488720128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574021.829694:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574021.829700:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574021.829705:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488720128:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574021.829718:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488720256:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574021.829747:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574021.829763:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574021/real 1788574021] req@ffff94d408c54380 x1875450488720256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574021.829779:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488720256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574021.829795:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574021.829799:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574021.829803:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488720256:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574021.829839:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488720384:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574021.829865:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574021.829880:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574021/real 1788574021] req@ffff94d408c54e00 x1875450488720384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574021.829895:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488720384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574021.829904:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574021.829907:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574021.829911:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488720384:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574021.829918:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488720512:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574021.829958:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574021.829976:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574021/real 1788574021] req@ffff94d408c56d80 x1875450488720512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574021.829989:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488720512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574021.829997:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574021.830000:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574021.830004:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488720512:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574021.830011:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488720640:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574021.830038:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574021.830050:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574021/real 1788574021] req@ffff94d408c54a80 x1875450488720640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574021.830063:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488720640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574021.830073:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574021.830076:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574021.830080:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488720640:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574022.341265:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574022.789420:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574023.365290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574023.813388:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574024.389283:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574024.837413:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574025.413840:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574025.861341:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574026.437349:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574026.885433:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574026.949279:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574026.949288:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574026.949291:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574026.949299:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5175 00000100:00080000:3.0:1788574026.949309:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574026.949315:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574026.949347:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488720768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574026.949370:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574026.949373:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574026.949374:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574026.949377:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5175 00000100:00080000:3.0:1788574026.949380:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574026.949383:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574026.949390:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574026.949399:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488720896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574026.949407:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488720768:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574026.949410:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574026.949417:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574026.949418:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574026.949422:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5175 00000100:00080000:3.0:1788574026.949424:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574026.949427:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574026.949430:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574026.949441:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488721024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574026.949449:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574026.949451:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574026.949453:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574026.949455:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5175 00000100:00080000:3.0:1788574026.949459:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574026.949462:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574026.949464:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574026.949471:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574026.949474:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488721152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574026.949482:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574026.949485:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574026.949486:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574026.949489:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5175 00000100:00080000:3.0:1788574026.949498:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574026.949501:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574026.949504:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574026.949512:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488721280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574026.949516:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574026/real 1788574026] req@ffff94d408c57b80 x1875450488720768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574026.949543:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488720768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574026.949556:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574026.949558:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574026.949561:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488720768:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574026.949580:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488720896:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574026.949598:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574026.949607:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574026/real 1788574026] req@ffff94d408c56680 x1875450488720896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574026.949620:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488720896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574026.949628:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574026.949641:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574026.949645:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488720896:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574026.949653:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488721024:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574026.949671:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574026.949679:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574026/real 1788574026] req@ffff94d408c54a80 x1875450488721024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574026.949687:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488721024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574026.949692:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574026.949695:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574026.949697:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488721024:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574026.949701:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488721152:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574026.949713:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574026.949718:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574026/real 1788574026] req@ffff94d408c56d80 x1875450488721152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574026.949725:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488721152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574026.949730:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574026.949732:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574026.949734:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488721152:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574026.949737:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488721280:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574026.949746:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574026.949752:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574026/real 1788574026] req@ffff94d408c54e00 x1875450488721280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574026.949758:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488721280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574026.949763:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574026.949765:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574026.949767:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488721280:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574027.461324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574027.909348:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574028.485605:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574028.933433:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574029.509582:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574029.957358:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574030.533367:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574030.981360:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574031.557351:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574032.005379:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574032.069462:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574032.069473:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574032.069479:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574032.069498:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5180 00000100:00080000:3.0:1788574032.069512:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574032.069522:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574032.069569:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488721408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574032.069608:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574032.069613:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574032.069616:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574032.069621:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5180 00000100:00080000:3.0:1788574032.069626:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574032.069632:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574032.069645:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574032.069665:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488721536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574032.069666:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488721408:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574032.069686:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574032.069691:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574032.069694:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574032.069699:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5180 00000100:00080000:3.0:1788574032.069704:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574032.069709:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574032.069715:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574032.069728:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488721664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574032.069745:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574032.069787:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574032/real 1788574032] req@ffff94d408c54380 x1875450488721408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574032.069811:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488721408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574032.069826:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574032.069832:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:3.0:1788574032.069833:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788574032.069837:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488721408:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574032.069841:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574032.069846:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574032.069853:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5180 00000100:00080000:3.0:1788574032.069858:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574032.069863:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574032.069870:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574032.069871:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488721536:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574032.069892:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488721792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574032.069906:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788574032.069907:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574032.069914:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574032.069916:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574032.069922:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5180 00000100:00080000:3.0:1788574032.069927:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788574032.069933:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574032/real 1788574032] req@ffff94d408c57100 x1875450488721536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574032.069934:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574032.069946:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574032.069951:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488721536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574032.069959:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574032.069963:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574032.069967:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488721920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574032.069968:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488721536:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574032.069973:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488721664:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574032.070005:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574032.070014:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574032/real 1788574032] req@ffff94d408c55180 x1875450488721664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574032.070022:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488721664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574032.070029:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574032.070031:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574032.070033:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488721664:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574032.070045:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488721792:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574032.070067:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574032.070074:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574032/real 1788574032] req@ffff94d408c57800 x1875450488721792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574032.070082:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488721792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574032.070089:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574032.070092:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574032.070094:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488721792:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574032.070307:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488721920:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574032.070335:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574032.070347:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574032/real 1788574032] req@ffff94d408c55c00 x1875450488721920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574032.070357:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488721920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574032.070365:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574032.070369:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574032.070372:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488721920:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574032.581252:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574033.029402:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574033.605273:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574034.053375:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574034.629266:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574035.077438:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574035.653309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574036.101348:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574036.677273:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574037.125387:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574037.189306:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574037.189316:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574037.189320:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574037.189331:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5185 00000100:00080000:0.0:1788574037.189341:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574037.189349:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574037.189388:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f800 x1875450488722048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574037.189417:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574037.189422:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574037.189425:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574037.189430:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5185 00000100:00080000:0.0:1788574037.189435:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574037.189440:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574037.189450:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574037.189450:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488722048:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574037.189469:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f100 x1875450488722176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574037.189482:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574037.189486:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574037.189489:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574037.189493:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5185 00000100:00080000:0.0:1788574037.189497:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574037.189502:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574037.189508:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788574037.189510:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574037.189526:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488722304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574037.189538:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574037.189541:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1788574037.189544:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574037/real 1788574037] req@ffff94d408d3f800 x1875450488722048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574037.189545:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574037.189556:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5185 00000100:00080000:0.0:1788574037.189562:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788574037.189567:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f800 x1875450488722048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574037.189570:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574037.189576:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788574037.189589:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788574037.189594:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488722432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574037.189596:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574037.189603:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488722048:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788574037.189616:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574037.189620:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574037.189622:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574037.189627:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488722176:192.168.202.125@tcp:38:kworker.0 00000100:00080000:0.0:1788574037.189629:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5185 00000100:00080000:0.0:1788574037.189635:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574037.189640:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574037.189646:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574037.189649:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574037.189661:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574037/real 1788574037] req@ffff94d408d3f100 x1875450488722176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574037.189666:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488722560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574037.189675:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f100 x1875450488722176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574037.189687:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574037.189691:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574037.189694:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488722176:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574037.189698:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488722304:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574037.189711:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574037.189725:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574037/real 1788574037] req@ffff94d408d3ca80 x1875450488722304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574037.189733:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488722304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574037.189742:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574037.189745:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574037.189747:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488722304:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574037.189750:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488722432:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574037.189762:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574037.189767:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574037/real 1788574037] req@ffff94d408d3d500 x1875450488722432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574037.189774:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488722432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574037.189780:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574037.189782:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574037.189784:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488722432:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574037.189793:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488722560:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574037.189802:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574037.189807:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574037/real 1788574037] req@ffff94d408d3e680 x1875450488722560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574037.189814:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488722560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574037.189819:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574037.189821:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574037.189823:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488722560:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574037.701261:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574038.149360:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574038.725314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574039.173396:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574039.749264:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574040.197356:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574040.773317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574041.221357:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574041.797287:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574042.245344:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574042.309314:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574042.309322:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574042.309325:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574042.309334:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5190 00000100:00080000:3.0:1788574042.309344:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574042.309350:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574042.309382:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488722688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574042.309404:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574042.309407:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574042.309409:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574042.309412:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5190 00000100:00080000:3.0:1788574042.309414:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574042.309416:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574042.309423:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574042.309441:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488722816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574042.309448:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488722688:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574042.309462:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574042.309466:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574042.309469:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574042.309474:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5190 00000100:00080000:3.0:1788574042.309478:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574042.309483:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574042.309489:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574042.309504:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488722944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574042.309511:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788574042.309515:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574042.309518:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574042.309520:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574042.309524:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5190 00000100:00080000:3.0:1788574042.309528:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574042.309532:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574042.309537:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574042.309548:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488723072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574042.309556:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574042/real 1788574042] req@ffff94d408c55880 x1875450488722688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574042.309557:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574042.309568:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574042.309572:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574042.309579:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5190 00000100:00100000:2.0:1788574042.309582:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488722688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574042.309587:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574042.309594:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574042.309597:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00080000:2.0:1788574042.309601:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574042.309605:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488722688:192.168.202.125@tcp:250:kworker.0 00000008:00100000:3.0:1788574042.309606:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574042.309621:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488723200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574042.309628:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488722816:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574042.309655:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574042.309665:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574042/real 1788574042] req@ffff94d408c56a00 x1875450488722816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574042.309674:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488722816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574042.309679:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574042.309681:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574042.309684:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488722816:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574042.309688:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488722944:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574042.309704:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574042.309712:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574042/real 1788574042] req@ffff94d408c57480 x1875450488722944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574042.309719:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488722944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574042.309725:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574042.309727:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574042.309729:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488722944:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574042.309733:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488723072:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574042.309744:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574042.309749:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574042/real 1788574042] req@ffff94d408c56300 x1875450488723072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574042.309757:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488723072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574042.309762:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574042.309763:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574042.309765:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488723072:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574042.309775:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488723200:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574042.309784:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574042.309790:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574042/real 1788574042] req@ffff94d408c54000 x1875450488723200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574042.309797:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488723200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574042.309802:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574042.309805:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574042.309807:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488723200:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574042.821345:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574043.269349:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574043.845301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574044.293409:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574044.869379:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574045.317425:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574045.893320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574046.341391:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574046.917333:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574047.365405:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574047.429347:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574047.429358:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574047.429362:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574047.429373:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5195 00000100:00080000:3.0:1788574047.429386:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574047.429394:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574047.429441:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488723328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574047.429468:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574047.429472:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574047.429473:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574047.429476:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5195 00000100:00080000:3.0:1788574047.429478:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574047.429480:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574047.429490:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574047.429499:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488723456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574047.429507:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574047.429510:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574047.429511:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574047.429514:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5195 00000100:00080000:3.0:1788574047.429516:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574047.429518:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574047.429521:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574047.429528:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488723328:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574047.429531:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488723584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574047.429543:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574047.429545:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574047.429546:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574047.429548:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5195 00000100:00080000:3.0:1788574047.429550:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574047.429552:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574047.429556:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574047.429564:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488723712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574047.429570:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574047.429572:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574047.429574:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574047.429577:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5195 00000100:00080000:3.0:1788574047.429597:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574047.429600:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574047.429603:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574047.429609:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574047.429613:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488723840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574047.429657:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574047/real 1788574047] req@ffff94d408c54700 x1875450488723328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574047.429674:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488723328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574047.429683:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574047.429686:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574047.429689:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488723328:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574047.429695:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488723456:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574047.429713:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574047.429721:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574047/real 1788574047] req@ffff94d408c55f80 x1875450488723456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574047.429729:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488723456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574047.429735:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574047.429737:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574047.429740:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488723456:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574047.429764:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488723584:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574047.429776:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574047.429783:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574047/real 1788574047] req@ffff94d408c54000 x1875450488723584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574047.429791:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488723584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574047.429797:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574047.429799:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574047.429802:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488723584:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574047.429805:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488723712:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574047.429815:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574047.429820:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574047/real 1788574047] req@ffff94d408c56300 x1875450488723712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574047.429828:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488723712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574047.429833:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574047.429835:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574047.429837:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488723712:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574047.429840:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488723840:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574047.429850:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574047.429855:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574047/real 1788574047] req@ffff94d408c57480 x1875450488723840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574047.429863:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488723840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574047.429868:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574047.429870:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574047.429872:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488723840:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574047.941338:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574048.389402:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574048.965330:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574049.413433:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574049.989338:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574050.437394:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574051.013354:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574051.461442:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574052.037309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574052.485380:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574052.549310:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574052.549319:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574052.549323:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574052.549331:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5200 00000100:00080000:3.0:1788574052.549341:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574052.549347:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574052.549379:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488723968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574052.549402:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574052.549405:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574052.549407:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574052.549409:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5200 00000100:00080000:3.0:1788574052.549412:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574052.549415:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574052.549423:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574052.549431:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488724096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574052.549438:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574052.549441:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574052.549443:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574052.549445:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5200 00000100:00080000:3.0:1788574052.549447:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574052.549449:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574052.549452:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574052.549458:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488724224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574052.549464:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488723968:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574052.549466:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574052.549473:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574052.549476:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574052.549480:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5200 00000100:00080000:3.0:1788574052.549483:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574052.549486:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574052.549492:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574052.549505:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488724352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574052.549516:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574052.549520:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574052.549523:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574052.549527:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5200 00000100:00080000:3.0:1788574052.549531:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788574052.549537:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574052.549540:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574052.549546:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574052.549557:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488724480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574052.549582:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574052/real 1788574052] req@ffff94d408c56a00 x1875450488723968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574052.549601:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488723968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574052.549613:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574052.549617:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574052.549622:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488723968:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574052.549629:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488724096:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574052.549652:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574052.549664:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574052/real 1788574052] req@ffff94d408c55880 x1875450488724096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574052.549676:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488724096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574052.549691:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574052.549694:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574052.549698:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488724096:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574052.549720:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488724224:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574052.549749:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574052.549761:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574052/real 1788574052] req@ffff94d408c55c00 x1875450488724224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574052.549773:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488724224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574052.549782:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574052.549786:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574052.549789:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488724224:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574052.549795:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488724352:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574052.549813:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574052.549823:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574052/real 1788574052] req@ffff94d408c57800 x1875450488724352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574052.549844:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488724352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574052.549853:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574052.549856:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574052.549861:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488724352:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574052.549867:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488724480:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574052.549887:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574052.549913:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574052/real 1788574052] req@ffff94d408c55180 x1875450488724480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574052.549926:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488724480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574052.549935:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574052.549938:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574052.549947:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488724480:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574053.061280:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574053.509367:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574054.085333:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574054.533459:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574055.109274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574055.557351:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574056.133341:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574056.581379:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574057.157276:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574057.605322:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574057.669305:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574057.669314:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574057.669318:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574057.669326:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5205 00000100:00080000:0.0:1788574057.669336:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574057.669342:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574057.669382:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488724608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574057.669403:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574057.669406:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574057.669407:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574057.669410:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5205 00000100:00080000:0.0:1788574057.669413:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574057.669415:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574057.669423:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574057.669432:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488724736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574057.669445:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574057.669448:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574057.669451:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574057.669452:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488724608:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788574057.669456:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5205 00000100:00080000:0.0:1788574057.669461:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574057.669464:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574057.669467:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574057.669478:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488724864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574057.669488:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574057.669492:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574057.669494:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574057.669499:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5205 00000100:00080000:0.0:1788574057.669503:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574057.669508:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574057.669515:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574057.669515:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574057.669533:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488724992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574057.669541:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574057.669543:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574057.669545:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574057.669549:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5205 00000100:00100000:2.0:1788574057.669553:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574057/real 1788574057] req@ffff94d408d3ea00 x1875450488724608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574057.669554:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574057.669561:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574057.669569:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574057.669576:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488724608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574057.669581:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488725120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574057.669595:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574057.669604:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574057.669610:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488724608:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574057.669631:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488724736:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574057.669647:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574057.669656:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574057/real 1788574057] req@ffff94d408d3ce00 x1875450488724736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574057.669664:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488724736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574057.669669:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574057.669672:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574057.669674:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488724736:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574057.669677:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488724864:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574057.669687:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574057.669693:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574057/real 1788574057] req@ffff94d408d3df80 x1875450488724864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574057.669701:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488724864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574057.669706:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574057.669707:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574057.669709:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488724864:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574057.669712:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488724992:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574057.669722:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574057.669728:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574057/real 1788574057] req@ffff94d408d3d180 x1875450488724992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574057.669735:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488724992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574057.669740:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574057.669742:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574057.669745:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488724992:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574057.669748:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488725120:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574057.669757:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574057.669762:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574057/real 1788574057] req@ffff94d408d3c000 x1875450488725120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574057.669769:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488725120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574057.669773:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574057.669775:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574057.669777:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488725120:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574058.181291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574058.629326:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574059.205375:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574059.653457:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574060.229325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574060.677405:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574061.253277:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574061.701328:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574062.277363:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574062.725370:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574062.789334:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574062.789346:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574062.789351:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574062.789362:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5211 00000100:00080000:3.0:1788574062.789375:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574062.789382:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574062.789446:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488725248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574062.789480:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574062.789483:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574062.789485:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574062.789488:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5211 00000100:00080000:3.0:1788574062.789492:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574062.789495:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574062.789505:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574062.789519:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488725376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574062.789527:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574062.789530:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574062.789531:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574062.789534:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5211 00000100:00080000:3.0:1788574062.789537:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788574062.789541:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488725248:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574062.789542:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574062.789546:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574062.789557:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488725504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574062.789564:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574062.789566:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574062.789567:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574062.789572:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5211 00000100:00080000:3.0:1788574062.789574:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574062.789577:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574062.789581:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574062.789594:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488725632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574062.789603:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574062.789607:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574062.789609:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574062.789613:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5211 00000100:00080000:3.0:1788574062.789617:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574062.789622:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:2.0:1788574062.789627:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:3.0:1788574062.789631:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574062.789645:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488725760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574062.789676:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574062/real 1788574062] req@ffff94d408c57100 x1875450488725248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574062.789697:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488725248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574062.789710:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574062.789715:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574062.789720:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488725248:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574062.789751:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488725376:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574062.789775:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574062.789790:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574062/real 1788574062] req@ffff94d408c54380 x1875450488725376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574062.789803:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488725376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574062.789812:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574062.789816:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574062.789837:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488725376:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574062.789844:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488725504:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574062.789867:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574062.789879:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574062/real 1788574062] req@ffff94d408c54e00 x1875450488725504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574062.789892:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488725504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574062.789901:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574062.789904:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574062.789909:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488725504:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574062.789915:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488725632:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574062.789936:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574062.789947:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574062/real 1788574062] req@ffff94d408c56d80 x1875450488725632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574062.789961:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488725632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574062.789970:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574062.789974:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574062.789977:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488725632:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574062.789984:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488725760:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574062.790004:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574062.790016:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574062/real 1788574062] req@ffff94d408c54a80 x1875450488725760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574062.790029:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488725760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574062.790037:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574062.790041:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574062.790044:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488725760:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574063.301291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574063.749347:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574064.325358:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574064.773381:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574065.349299:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574065.797388:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574066.373351:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574066.821386:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574067.397343:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574067.845367:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574067.909319:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574067.909331:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574067.909335:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574067.909347:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5216 00000100:00080000:3.0:1788574067.909360:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574067.909369:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574067.909412:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488725888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574067.909448:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574067.909452:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574067.909454:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574067.909459:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5216 00000100:00080000:3.0:1788574067.909464:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574067.909470:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574067.909482:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574067.909496:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488725888:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574067.909501:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488726016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574067.909511:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574067.909515:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574067.909516:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574067.909520:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5216 00000100:00080000:3.0:1788574067.909524:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574067.909528:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574067.909532:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574067.909544:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488726144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574067.909551:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574067.909554:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574067.909556:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574067.909559:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5216 00000100:00080000:3.0:1788574067.909563:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574067.909565:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574067.909569:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574067.909574:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574067.909579:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488726272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574067.909588:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574067.909590:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574067.909592:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574067.909595:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5216 00000100:00080000:3.0:1788574067.909597:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574067.909600:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574067.909612:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574067/real 1788574067] req@ffff94d408c56680 x1875450488725888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574067.909631:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488725888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574067.909646:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574067.909651:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574067.909655:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488725888:192.168.202.125@tcp:250:kworker.0 00000008:00100000:3.0:1788574067.909669:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574067.909684:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488726016:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574067.909685:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488726400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574067.909709:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574067.909720:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574067/real 1788574067] req@ffff94d408c57b80 x1875450488726016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574067.909730:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488726016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574067.909738:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574067.909742:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574067.909744:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488726016:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574067.909750:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488726144:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574067.909771:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574067.909798:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574067/real 1788574067] req@ffff94d408c54a80 x1875450488726144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574067.909812:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488726144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574067.909822:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574067.909826:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574067.909830:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488726144:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574067.909837:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488726272:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574067.909860:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574067.909874:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574067/real 1788574067] req@ffff94d408c56d80 x1875450488726272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574067.909889:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488726272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574067.909899:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574067.909905:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574067.909910:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488726272:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574067.909935:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488726400:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574067.909958:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574067.909971:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574067/real 1788574067] req@ffff94d408c54e00 x1875450488726400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574067.909981:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488726400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574067.909987:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574067.909989:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574067.909991:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488726400:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574068.421351:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574068.869433:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574069.445303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574069.893368:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574070.469305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574070.917377:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574071.493431:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574071.941416:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574072.517354:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574072.965411:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574073.029303:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574073.029311:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574073.029320:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574073.029328:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5221 00000100:00080000:3.0:1788574073.029337:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574073.029344:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574073.029377:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488726528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574073.029399:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574073.029401:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574073.029403:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574073.029405:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5221 00000100:00080000:3.0:1788574073.029407:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574073.029410:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574073.029417:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574073.029426:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488726656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574073.029433:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574073.029436:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574073.029437:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574073.029440:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5221 00000100:00080000:3.0:1788574073.029441:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574073.029444:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574073.029447:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574073.029457:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488726528:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574073.029457:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488726784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574073.029479:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574073.029487:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574073.029489:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574073.029493:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5221 00000100:00080000:3.0:1788574073.029495:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574073.029499:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574073.029503:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574073.029513:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488726912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574073.029521:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574073.029524:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574073.029525:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574073.029527:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5221 00000100:00080000:3.0:1788574073.029539:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574073.029542:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574073.029545:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574073.029554:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574073.029555:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488727040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574073.029603:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574073/real 1788574073] req@ffff94d408c54380 x1875450488726528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574073.029621:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488726528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574073.029630:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574073.029634:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574073.029637:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488726528:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574073.029644:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488726656:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574073.029661:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574073.029670:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574073/real 1788574073] req@ffff94d408c57100 x1875450488726656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574073.029679:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488726656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574073.029685:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574073.029687:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574073.029689:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488726656:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574073.029713:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488726784:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574073.029725:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574073.029731:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574073/real 1788574073] req@ffff94d408c55180 x1875450488726784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574073.029761:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488726784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574073.029774:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574073.029783:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574073.029786:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488726784:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574073.029791:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488726912:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574073.029808:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574073.029819:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574073/real 1788574073] req@ffff94d408c57800 x1875450488726912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574073.029832:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488726912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574073.029841:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574073.029845:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574073.029849:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488726912:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574073.029856:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488727040:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574073.029877:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574073.029888:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574073/real 1788574073] req@ffff94d408c55c00 x1875450488727040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574073.029897:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488727040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574073.029902:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574073.029905:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574073.029908:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488727040:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574073.541299:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574073.989445:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574074.565300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574075.013390:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574075.589286:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574076.037374:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574076.613266:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574077.061315:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574077.637357:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574078.085368:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574078.149348:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574078.149355:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574078.149358:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574078.149366:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5226 00000100:00080000:2.0:1788574078.149381:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574078.149387:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574078.149420:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488727168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574078.149444:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574078.149447:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574078.149448:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574078.149451:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5226 00000100:00080000:2.0:1788574078.149453:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574078.149456:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574078.149463:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574078.149471:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488727296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574078.149479:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788574078.149484:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488727168:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788574078.149484:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574078.149486:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574078.149494:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5226 00000100:00080000:2.0:1788574078.149497:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574078.149500:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574078.149502:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574078.149511:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488727424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574078.149518:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574078.149520:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574078.149521:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574078.149524:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5226 00000100:00080000:2.0:1788574078.149527:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574078.149529:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574078.149532:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574078.149539:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488727552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574078.149546:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788574078.149546:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574078.149551:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574078.149553:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574078.149557:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5226 00000100:00080000:2.0:1788574078.149561:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574078.149563:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574078.149566:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574078.149574:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488727680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574078.149584:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574078/real 1788574078] req@ffff94d408c63480 x1875450488727168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574078.149600:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488727168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574078.149608:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574078.149611:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574078.149613:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488727168:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574078.149632:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488727296:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574078.149652:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574078.149666:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574078/real 1788574078] req@ffff94d408c61f80 x1875450488727296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574078.149681:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488727296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574078.149690:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574078.149694:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574078.149697:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488727296:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574078.149713:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488727424:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574078.149729:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574078.149738:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574078/real 1788574078] req@ffff94d408c63b80 x1875450488727424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574078.149746:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488727424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574078.149752:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574078.149754:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574078.149756:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488727424:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574078.149760:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488727552:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574078.149771:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574078.149777:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574078/real 1788574078] req@ffff94d408c61500 x1875450488727552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574078.149784:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488727552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574078.149789:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574078.149791:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574078.149793:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488727552:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574078.149796:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488727680:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574078.149805:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574078.149812:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574078/real 1788574078] req@ffff94d408c61c00 x1875450488727680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574078.149819:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488727680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574078.149824:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574078.149826:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574078.149827:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488727680:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574078.661347:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574079.109371:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574079.685351:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574080.133390:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574080.709303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574081.157421:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574081.733339:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574082.181355:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574082.757349:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574083.205366:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574083.269298:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574083.269309:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574083.269313:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574083.269324:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5231 00000100:00080000:0.0:1788574083.269337:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574083.269345:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574083.269389:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488727808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574083.269418:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574083.269422:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574083.269423:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574083.269427:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5231 00000100:00080000:0.0:1788574083.269430:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574083.269433:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574083.269443:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574083.269456:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488727936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574083.269456:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488727808:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788574083.269472:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574083.269475:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574083.269476:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574083.269479:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5231 00000100:00080000:0.0:1788574083.269482:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574083.269484:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574083.269487:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574083.269495:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488728064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574083.269502:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574083.269504:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574083.269506:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574083.269509:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5231 00000100:00080000:0.0:1788574083.269512:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574083.269515:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574083.269518:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574083.269525:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488728192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574083.269531:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574083.269534:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574083.269535:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574083.269538:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5231 00000100:00080000:0.0:1788574083.269541:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574083.269543:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574083.269547:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574083.269555:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488728320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574083.269574:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488727936:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574083.269603:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488728064:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574083.269620:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488728192:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574083.269638:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488728320:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574083.781377:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574084.229379:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574084.805264:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574085.253334:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574085.829284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574086.277397:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788574086.341487:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:3.0:1788574086.341534:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788574086.341551:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:3.0:1788574086.341594:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:3.0:1788574086.341606:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:3.0:1788574086.341615:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:3.0:1788574086.341623:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:2.0:1788574086.341680:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574083/real 1788574086] req@ffff94d408d3c380 x1875450488727936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574086.341707:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488727936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574086.341719:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574086.341724:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574086.341728:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488727936:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574086.341736:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574083/real 1788574086] req@ffff94d408d3f480 x1875450488728064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574086.341747:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488728064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574086.341755:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574086.341759:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574086.341762:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488728064:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574086.341769:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574083/real 1788574086] req@ffff94d408d3c700 x1875450488728192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574086.341795:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488728192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574086.341803:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574086.341806:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574086.341809:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488728192:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574086.341817:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574083/real 1788574086] req@ffff94d408d3e300 x1875450488728320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574086.341828:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488728320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574086.341836:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574086.341840:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574086.341843:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488728320:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574086.341850:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574083/real 1788574086] req@ffff94d408d3dc00 x1875450488727808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574086.341861:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488727808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574086.341868:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574086.341871:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574086.341875:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488727808:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574086.853295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574087.301343:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574087.877258:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574088.325402:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574088.389398:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574088.389410:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574088.389414:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574088.389426:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5236 00000100:00080000:3.0:1788574088.389438:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574088.389447:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574088.389488:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488728448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574088.389518:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574088.389521:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574088.389522:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574088.389525:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5236 00000100:00080000:3.0:1788574088.389528:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574088.389530:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574088.389540:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574088.389558:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488728576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574088.389569:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574088.389573:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574088.389575:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574088.389579:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5236 00000100:00080000:3.0:1788574088.389583:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574088.389587:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574088.389593:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574088.389605:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488728704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574088.389616:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574088.389620:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574088.389636:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574088.389639:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488728448:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574088.389642:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5236 00000100:00080000:3.0:1788574088.389651:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574088.389658:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574088.389664:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574088.389679:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488728832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574088.389689:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574088.389693:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574088.389695:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574088.389699:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5236 00000100:00080000:3.0:1788574088.389726:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574088.389731:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574088.389737:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574088.389741:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574088.389755:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488728960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574088.389797:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574088/real 1788574088] req@ffff94d408c55880 x1875450488728448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574088.389821:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488728448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574088.389833:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574088.389838:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574088.389843:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488728448:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574088.389851:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488728576:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574088.389878:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574088.389893:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574088/real 1788574088] req@ffff94d408c56a00 x1875450488728576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574088.389907:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488728576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574088.389915:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574088.389920:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574088.389923:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488728576:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574088.389970:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488728704:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574088.389999:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574088.390015:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574088/real 1788574088] req@ffff94d408c57480 x1875450488728704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574088.390035:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488728704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574088.390043:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574088.390047:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574088.390051:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488728704:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574088.390058:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488728832:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574088.390080:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574088.390091:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574088/real 1788574088] req@ffff94d408c56300 x1875450488728832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574088.390105:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488728832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574088.390113:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574088.390117:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574088.390120:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488728832:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574088.390126:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488728960:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574088.390147:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574088.390160:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574088/real 1788574088] req@ffff94d408c54000 x1875450488728960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574088.390171:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488728960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574088.390185:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574088.390188:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574088.390192:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488728960:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574088.901286:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574089.349372:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574089.925289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574090.373349:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574090.949285:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574091.397331:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574091.973966:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574092.421402:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574092.997326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574093.445374:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574093.509324:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574093.509336:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574093.509341:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574093.509352:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5241 00000100:00080000:3.0:1788574093.509365:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574093.509374:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574093.509421:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488729088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574093.509450:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574093.509453:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574093.509455:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574093.509458:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5241 00000100:00080000:3.0:1788574093.509460:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574093.509462:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574093.509472:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574093.509481:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488729216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574093.509491:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574093.509494:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574093.509496:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574093.509501:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5241 00000100:00080000:3.0:1788574093.509505:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788574093.509509:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488729088:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574093.509511:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574093.509520:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574093.509538:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488729344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574093.509545:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574093.509548:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574093.509550:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574093.509553:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5241 00000100:00080000:3.0:1788574093.509557:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574093.509560:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574093.509567:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574093.509575:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488729472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574093.509581:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788574093.509603:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574093.509604:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574093.509614:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574093.509620:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5241 00000100:00080000:3.0:1788574093.509624:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574093.509629:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574093.509635:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574093.509646:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488729600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574093.509656:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574093/real 1788574093] req@ffff94d408c55f80 x1875450488729088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574093.509673:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488729088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574093.509683:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574093.509687:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574093.509690:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488729088:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574093.509711:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488729216:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574093.509730:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574093.509738:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574093/real 1788574093] req@ffff94d408c54700 x1875450488729216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574093.509746:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488729216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574093.509752:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574093.509755:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574093.509757:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488729216:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574093.509761:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488729344:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574093.509773:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574093.509778:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574093/real 1788574093] req@ffff94d408c54000 x1875450488729344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574093.509786:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488729344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574093.509791:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574093.509793:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574093.509795:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488729344:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574093.509799:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488729472:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574093.509810:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574093.509815:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574093/real 1788574093] req@ffff94d408c56300 x1875450488729472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574093.509823:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488729472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574093.509828:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574093.509829:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574093.509831:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488729472:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574093.509835:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488729600:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574093.509844:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574093.509849:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574093/real 1788574093] req@ffff94d408c57480 x1875450488729600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574093.509857:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488729600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574093.509862:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574093.509864:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574093.509865:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488729600:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574094.021615:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574094.469398:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574095.045775:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574095.493376:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574096.069323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574096.517348:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574097.093833:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574097.541386:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574098.117263:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574098.565376:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574098.629293:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574098.629305:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574098.629310:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574098.629321:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5246 00000100:00080000:3.0:1788574098.629334:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574098.629343:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574098.629387:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488729728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574098.629414:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574098.629417:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574098.629419:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574098.629422:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5246 00000100:00080000:3.0:1788574098.629424:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574098.629427:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574098.629436:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574098.629446:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488729856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574098.629455:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574098.629461:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574098.629463:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574098.629472:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488729728:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574098.629473:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5246 00000100:00080000:3.0:1788574098.629488:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574098.629498:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574098.629505:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574098.629524:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488729984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574098.629554:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574098.629559:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574098.629562:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574098.629567:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5246 00000100:00080000:3.0:1788574098.629573:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788574098.629574:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574098.629587:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574098.629594:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574098.629612:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488730112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574098.629624:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574098.629626:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574098/real 1788574098] req@ffff94d408c56a00 x1875450488729728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574098.629632:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574098.629638:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574098.629651:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5246 00000100:00080000:3.0:1788574098.629659:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574098.629665:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574098.629674:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488729728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788574098.629677:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788574098.629699:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788574098.629705:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488730240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574098.629707:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574098.629716:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488729728:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574098.629756:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488729856:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574098.629781:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574098.629790:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574098/real 1788574098] req@ffff94d408c55880 x1875450488729856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574098.629799:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488729856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574098.629809:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574098.629811:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574098.629814:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488729856:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574098.629818:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488729984:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574098.629829:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574098.629835:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574098/real 1788574098] req@ffff94d408c55c00 x1875450488729984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574098.629842:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488729984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574098.629848:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574098.629850:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574098.629852:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488729984:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574098.629856:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488730112:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574098.629867:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574098.629873:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574098/real 1788574098] req@ffff94d408c57800 x1875450488730112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574098.629880:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488730112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574098.629885:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574098.629887:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574098.629889:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488730112:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574098.629904:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488730240:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574098.629916:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574098.629923:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574098/real 1788574098] req@ffff94d408c55180 x1875450488730240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574098.629930:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488730240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574098.629935:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574098.629937:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574098.629939:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488730240:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574099.141341:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574099.589375:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574100.165353:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574100.613339:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574101.189295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574101.637385:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574102.213261:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574102.661355:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574103.237286:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574103.685345:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574103.749309:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574103.749318:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574103.749321:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574103.749329:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5252 00000100:00080000:0.0:1788574103.749338:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574103.749345:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574103.749375:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488730368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574103.749407:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574103.749410:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574103.749412:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574103.749415:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5252 00000100:00080000:0.0:1788574103.749418:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574103.749423:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574103.749431:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574103.749446:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488730368:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574103.749450:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ed80 x1875450488730496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574103.749462:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574103.749466:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574103.749467:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574103.749472:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5252 00000100:00080000:0.0:1788574103.749474:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574103.749478:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574103.749482:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574103.749493:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488730624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574103.749524:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574103.749529:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1788574103.749530:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788574103.749534:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574103.749543:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5252 00000100:00080000:0.0:1788574103.749548:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574103.749553:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574103.749559:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574103.749572:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488730752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574103.749583:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574103.749587:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574103.749589:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574103.749594:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5252 00000100:00080000:0.0:1788574103.749598:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574103.749603:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574103.749607:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574103.749620:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574103/real 1788574103] req@ffff94d408d3d880 x1875450488730368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574103.749621:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488730880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574103.749649:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488730368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574103.749662:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574103.749666:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574103.749671:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488730368:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574103.749695:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488730496:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574103.749832:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574103.749850:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574103/real 1788574103] req@ffff94d408d3ed80 x1875450488730496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574103.749865:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ed80 x1875450488730496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574103.749914:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574103.749921:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574103.749927:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488730496:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574103.749936:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488730624:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574103.749961:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574103.749974:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574103/real 1788574103] req@ffff94d408d3dc00 x1875450488730624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574103.749987:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488730624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574103.749995:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574103.749999:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574103.750002:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488730624:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574103.750009:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488730752:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574103.750027:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574103.750037:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574103/real 1788574103] req@ffff94d408d3e300 x1875450488730752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574103.750049:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488730752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574103.750057:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574103.750061:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574103.750064:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488730752:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574103.750070:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488730880:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574103.750086:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574103.750096:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574103/real 1788574103] req@ffff94d408d3c700 x1875450488730880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574103.750108:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488730880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574103.750116:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574103.750119:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574103.750123:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488730880:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574104.261272:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574104.709344:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574105.285315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574105.733386:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574106.309307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574106.757433:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574107.333386:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574107.781376:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574108.357298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574108.805402:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574108.869272:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574108.869283:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574108.869287:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574108.869297:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5257 00000100:00080000:0.0:1788574108.869310:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574108.869318:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574108.869386:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488731008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574108.869416:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574108.869419:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574108.869421:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574108.869424:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5257 00000100:00080000:0.0:1788574108.869426:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574108.869429:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574108.869438:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574108.869447:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488731136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574108.869455:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574108.869458:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574108.869459:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574108.869463:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5257 00000100:00080000:0.0:1788574108.869464:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574108.869467:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574108.869470:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574108.869477:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488731264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574108.869484:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574108.869487:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574108.869490:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574108.869492:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488731008:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788574108.869494:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5257 00000100:00080000:0.0:1788574108.869500:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574108.869503:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574108.869507:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574108.869514:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488731392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574108.869520:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574108.869523:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574108.869524:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574108.869527:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5257 00000100:00080000:0.0:1788574108.869529:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574108.869532:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574108.869535:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574108.869544:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488731520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574108.869582:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574108.869620:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574108/real 1788574108] req@ffff94d408d3f480 x1875450488731008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574108.869638:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488731008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574108.869647:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574108.869651:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574108.869654:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488731008:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574108.869660:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488731136:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574108.869675:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574108.869683:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574108/real 1788574108] req@ffff94d408d3c380 x1875450488731136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574108.869696:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488731136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574108.869702:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574108.869704:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574108.869706:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488731136:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574108.869727:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488731264:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574108.869738:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574108.869743:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574108/real 1788574108] req@ffff94d408d3c000 x1875450488731264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574108.869751:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488731264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574108.869756:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574108.869758:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574108.869760:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488731264:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574108.869764:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488731392:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574108.869774:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574108.869779:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574108/real 1788574108] req@ffff94d408d3d180 x1875450488731392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574108.869787:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488731392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574108.869796:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574108.869798:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574108.869801:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488731392:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574108.869804:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488731520:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574108.869813:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574108.869818:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574108/real 1788574108] req@ffff94d408d3df80 x1875450488731520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574108.869826:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488731520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574108.869831:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574108.869832:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574108.869834:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488731520:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574109.381289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574109.829467:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574110.405293:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574110.853341:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574111.429344:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574111.877432:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574112.453302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574112.901391:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574113.477304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574113.925342:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574113.989302:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574113.989309:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574113.989312:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574113.989321:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5262 00000100:00080000:0.0:1788574113.989330:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574113.989337:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574113.989370:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488731648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574113.989394:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574113.989397:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574113.989398:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574113.989401:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5262 00000100:00080000:0.0:1788574113.989403:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574113.989405:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574113.989412:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574113.989421:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488731776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574113.989429:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574113.989431:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574113.989433:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574113.989435:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5262 00000100:00080000:0.0:1788574113.989437:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574113.989439:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574113.989442:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574113.989446:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488731648:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574113.989452:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488731904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574113.989460:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574113.989463:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574113.989464:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574113.989467:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5262 00000100:00080000:0.0:1788574113.989469:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574113.989471:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574113.989474:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574113.989483:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488732032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574113.989490:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574113.989492:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574113.989493:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574113.989495:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5262 00000100:00080000:0.0:1788574113.989498:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574113.989501:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574113.989503:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574113.989511:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488732160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574113.989518:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574113.989550:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574113/real 1788574113] req@ffff94d408d3ce00 x1875450488731648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574113.989568:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488731648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574113.989575:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574113.989578:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574113.989581:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488731648:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574113.989586:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488731776:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574113.989600:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574113.989608:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574113/real 1788574113] req@ffff94d408d3ea00 x1875450488731776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574113.989616:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488731776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574113.989621:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574113.989623:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574113.989626:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488731776:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574113.989643:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488731904:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574113.989653:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574113.989659:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574113/real 1788574113] req@ffff94d408d3e680 x1875450488731904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574113.989666:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488731904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574113.989676:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574113.989678:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574113.989680:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488731904:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574113.989683:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488732032:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574113.989693:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574113.989698:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574113/real 1788574113] req@ffff94d408d3d500 x1875450488732032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574113.989705:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488732032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574113.989710:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574113.989712:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574113.989713:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488732032:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574113.989717:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488732160:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574113.989725:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574113.989731:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574113/real 1788574113] req@ffff94d408d3ca80 x1875450488732160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574113.989738:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488732160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574113.989743:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574113.989745:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574113.989747:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488732160:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574114.501274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574114.949396:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574115.525297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574115.973334:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574116.549355:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574116.997411:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574117.573298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574118.021357:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574118.597284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574119.045365:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574119.109326:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574119.109338:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574119.109342:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574119.109353:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5267 00000100:00080000:3.0:1788574119.109362:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574119.109368:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574119.109404:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488732288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574119.109435:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574119.109439:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574119.109441:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574119.109450:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5267 00000100:00080000:3.0:1788574119.109454:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574119.109458:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574119.109468:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574119.109483:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488732416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574119.109490:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488732288:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574119.109509:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574119.109512:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574119.109514:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574119.109518:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5267 00000100:00080000:3.0:1788574119.109520:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574119.109524:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574119.109528:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574119.109538:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488732544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574119.109547:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574119.109551:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574119.109553:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574119.109556:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5267 00000100:00080000:3.0:1788574119.109561:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788574119.109562:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574119.109571:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574119.109578:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574119.109592:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488732672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574119.109602:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574119.109603:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574119/real 1788574119] req@ffff94d408c57100 x1875450488732288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574119.109605:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574119.109608:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574119.109615:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5267 00000100:00080000:3.0:1788574119.109621:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574119.109627:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574119.109631:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574119.109639:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488732288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574119.109641:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488732800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574119.109674:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574119.109686:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574119.109692:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488732288:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574119.109711:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488732416:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574119.109730:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574119.109740:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574119/real 1788574119] req@ffff94d408c54380 x1875450488732416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574119.109749:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488732416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574119.109755:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574119.109757:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574119.109760:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488732416:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574119.109763:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488732544:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574119.109774:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574119.109780:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574119/real 1788574119] req@ffff94d408c54e00 x1875450488732544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574119.109788:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488732544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574119.109793:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574119.109795:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574119.109797:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488732544:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574119.109800:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488732672:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574119.109810:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574119.109816:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574119/real 1788574119] req@ffff94d408c56d80 x1875450488732672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574119.109823:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488732672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574119.109829:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574119.109831:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574119.109833:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488732672:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574119.109836:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488732800:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574119.109846:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574119.109851:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574119/real 1788574119] req@ffff94d408c54a80 x1875450488732800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574119.109858:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488732800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574119.109862:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574119.109865:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574119.109867:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488732800:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574119.621284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574120.069334:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574120.645319:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574121.093438:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574121.669303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574122.117395:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574122.693292:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574123.141340:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574123.717283:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574124.165327:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574124.229284:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574124.229292:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574124.229295:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574124.229304:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5272 00000100:00080000:3.0:1788574124.229313:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574124.229319:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574124.229353:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488732928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574124.229376:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574124.229379:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574124.229381:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574124.229383:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5272 00000100:00080000:3.0:1788574124.229386:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574124.229388:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574124.229396:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574124.229405:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488733056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574124.229414:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574124.229416:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574124.229418:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574124.229420:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5272 00000100:00080000:3.0:1788574124.229422:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788574124.229426:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488732928:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574124.229427:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574124.229434:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574124.229444:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488733184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574124.229451:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574124.229454:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574124.229456:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574124.229458:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5272 00000100:00080000:3.0:1788574124.229460:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574124.229463:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574124.229467:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574124.229473:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488733312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574124.229480:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574124.229487:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574124.229489:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574124.229494:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5272 00000800:00000100:2.0:1788574124.229495:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574124.229498:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574124.229502:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574124.229506:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574124.229514:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488733440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574124.229534:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574124/real 1788574124] req@ffff94d408c57b80 x1875450488732928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574124.229548:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488732928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574124.229555:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574124.229558:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574124.229561:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488732928:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574124.229566:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488733056:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574124.229582:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574124.229589:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574124/real 1788574124] req@ffff94d408c56680 x1875450488733056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574124.229597:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488733056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574124.229603:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574124.229605:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574124.229606:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488733056:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574124.229625:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488733184:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574124.229635:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574124.229641:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574124/real 1788574124] req@ffff94d408c54a80 x1875450488733184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574124.229648:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488733184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574124.229654:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574124.229656:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574124.229657:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488733184:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574124.229661:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488733312:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574124.229670:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574124.229676:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574124/real 1788574124] req@ffff94d408c56d80 x1875450488733312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574124.229683:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488733312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574124.229688:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574124.229690:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574124.229692:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488733312:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574124.229695:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488733440:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574124.229704:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574124.229709:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574124/real 1788574124] req@ffff94d408c54e00 x1875450488733440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574124.229716:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488733440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574124.229728:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574124.229730:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574124.229733:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488733440:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574124.741291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574125.189342:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574125.765274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574126.213345:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574126.789307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574127.237363:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574127.813300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574128.261337:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574128.837302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574129.285358:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574129.349320:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574129.349333:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574129.349338:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574129.349349:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5277 00000100:00080000:3.0:1788574129.349361:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574129.349369:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574129.349414:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488733568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574129.349447:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574129.349450:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574129.349452:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574129.349455:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5277 00000100:00080000:3.0:1788574129.349458:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574129.349462:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574129.349471:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574129.349482:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488733696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574129.349491:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574129.349494:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574129.349497:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574129.349506:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5277 00000100:00100000:2.0:1788574129.349507:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488733568:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574129.349509:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574129.349519:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574129.349526:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574129.349537:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488733824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574129.349547:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574129.349550:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574129.349551:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574129.349554:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5277 00000100:00080000:3.0:1788574129.349557:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574129.349560:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574129.349564:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574129.349572:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488733952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574129.349579:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574129.349582:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574129.349583:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574129.349586:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5277 00000100:00080000:3.0:1788574129.349589:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574129.349594:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:2.0:1788574129.349595:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:3.0:1788574129.349607:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574129.349620:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488734080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574129.349648:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574129/real 1788574129] req@ffff94d408c54380 x1875450488733568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574129.349666:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488733568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574129.349679:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574129.349683:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574129.349686:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488733568:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574129.349706:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488733696:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574129.349735:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574129.349749:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574129/real 1788574129] req@ffff94d408c57100 x1875450488733696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574129.349758:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488733696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574129.349763:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574129.349766:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574129.349768:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488733696:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574129.349772:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488733824:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574129.349786:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574129.349792:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574129/real 1788574129] req@ffff94d408c55180 x1875450488733824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574129.349800:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488733824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574129.349805:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574129.349807:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574129.349817:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488733824:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574129.349821:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488733952:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574129.349833:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574129.349840:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574129/real 1788574129] req@ffff94d408c57800 x1875450488733952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574129.349847:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488733952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574129.349852:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574129.349855:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574129.349857:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488733952:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574129.349861:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488734080:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574129.349871:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574129.349876:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574129/real 1788574129] req@ffff94d408c55c00 x1875450488734080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574129.349882:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488734080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574129.349888:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574129.349890:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574129.349891:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488734080:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574129.861315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574130.309362:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574130.885347:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788574131.333419:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574131.909282:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574132.357345:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574132.933354:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574133.381418:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574133.957371:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574134.405457:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574134.469328:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574134.469341:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574134.469346:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574134.469357:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5282 00000100:00080000:2.0:1788574134.469370:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574134.469378:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574134.469428:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488734208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574134.469465:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574134.469468:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574134.469470:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574134.469473:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5282 00000100:00080000:2.0:1788574134.469478:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574134.469482:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574134.469492:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574134.469503:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63100 x1875450488734336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574134.469515:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488734208:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788574134.469519:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574134.469532:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574134.469534:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574134.469538:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5282 00000100:00080000:2.0:1788574134.469541:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574134.469544:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574134.469548:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574134.469557:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60e00 x1875450488734464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574134.469583:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574134.469585:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574134.469587:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574134.469592:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5282 00000100:00080000:2.0:1788574134.469605:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574134.469609:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574134.469613:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574134.469616:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574134.469632:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488734592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574134.469641:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574134.469649:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574134.469652:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574134.469655:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5282 00000100:00080000:2.0:1788574134.469659:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574134.469665:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574134.469665:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574134/real 1788574134] req@ffff94d408c61880 x1875450488734208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788574134.469669:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574134.469687:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488734720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574134.469716:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488734208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574134.469727:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574134.469732:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574134.469737:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488734208:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574134.469759:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488734336:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574134.469780:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574134.469789:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574134/real 1788574134] req@ffff94d408c63100 x1875450488734336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574134.469798:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63100 x1875450488734336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574134.469804:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574134.469807:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574134.469809:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488734336:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574134.469813:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488734464:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574134.469826:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574134.469831:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574134/real 1788574134] req@ffff94d408c60e00 x1875450488734464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574134.469839:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60e00 x1875450488734464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574134.469845:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574134.469847:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574134.469848:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488734464:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574134.469852:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488734592:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574134.469862:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574134.469868:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574134/real 1788574134] req@ffff94d408c61c00 x1875450488734592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574134.469881:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488734592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574134.469886:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574134.469888:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574134.469889:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488734592:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574134.469893:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488734720:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574134.469902:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574134.469908:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574134/real 1788574134] req@ffff94d408c61500 x1875450488734720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574134.469915:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488734720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574134.469921:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574134.469922:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574134.469924:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488734720:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574134.981300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574135.429361:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574136.005337:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574136.453433:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574137.029280:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574137.477383:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574138.053364:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574138.501436:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574139.077316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574139.525405:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574139.589367:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574139.589377:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574139.589381:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574139.589389:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5287 00000100:00080000:3.0:1788574139.589399:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574139.589421:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574139.589465:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488734848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574139.589496:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574139.589500:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574139.589502:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574139.589504:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5287 00000100:00080000:3.0:1788574139.589507:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574139.589510:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574139.589521:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574139.589533:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488734976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574139.589541:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574139.589544:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574139.589547:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574139.589551:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488734848:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574139.589553:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5287 00000100:00080000:3.0:1788574139.589565:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574139.589571:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574139.589575:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574139.589594:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488735104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574139.589601:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574139.589604:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574139.589606:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574139.589609:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5287 00000100:00080000:3.0:1788574139.589630:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788574139.589642:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574139.589682:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574139/real 1788574139] req@ffff94d408c55880 x1875450488734848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574139.589704:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488734848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574139.589714:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574139.589720:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574139.589724:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488734848:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574139.589728:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574139.589736:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574139.589752:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488734976:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574139.589771:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574139.589780:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574139/real 1788574139] req@ffff94d408c56a00 x1875450488734976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574139.589789:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488734976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574139.589796:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574139.589799:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574139.589801:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488734976:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574139.589806:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488735104:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574139.589817:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574139.589826:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574139/real 1788574139] req@ffff94d408c57480 x1875450488735104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574139.589834:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488735104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574139.589845:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574139.589850:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574139.589853:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488735104:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574139.589874:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488735232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574139.589930:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574139.589934:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574139.589936:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574139.589945:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5287 00000100:00080000:3.0:1788574139.589951:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574139.589956:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574139.589964:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574139.589966:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488735232:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574139.589982:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488735360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574139.590006:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574139.590034:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574139/real 1788574139] req@ffff94d408c56300 x1875450488735232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574139.590056:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488735232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574139.590067:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574139.590071:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574139.590074:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488735232:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574139.590086:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488735360:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574139.590103:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574139.590110:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574139/real 1788574139] req@ffff94d408c54000 x1875450488735360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574139.590119:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488735360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574139.590124:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574139.590126:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574139.590129:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488735360:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574140.101349:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574140.549435:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574141.125359:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574141.573402:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574142.149306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574142.597449:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574143.173403:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574143.621373:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574144.197359:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574144.645386:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574144.709337:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574144.709345:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574144.709348:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574144.709357:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5293 00000100:00080000:0.0:1788574144.709366:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574144.709373:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574144.709405:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f100 x1875450488735488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574144.709429:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574144.709432:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574144.709433:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574144.709437:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5293 00000100:00080000:0.0:1788574144.709445:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574144.709449:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574144.709457:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574144.709463:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488735488:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574144.709470:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f800 x1875450488735616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574144.709479:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574144.709481:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574144.709483:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574144.709485:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5293 00000100:00080000:0.0:1788574144.709488:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574144.709490:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574144.709493:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574144.709503:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488735744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574144.709509:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574144.709512:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574144.709514:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574144.709516:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5293 00000100:00080000:0.0:1788574144.709518:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574144.709521:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:2.0:1788574144.709526:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:0.0:1788574144.709528:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574144.709538:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488735872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574144.709545:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574144.709548:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574144.709549:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574144.709553:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5293 00000100:00080000:0.0:1788574144.709556:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574144.709561:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574144.709561:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574144/real 1788574144] req@ffff94d408d3f100 x1875450488735488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788574144.709564:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574144.709575:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488736000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574144.709583:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f100 x1875450488735488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574144.709592:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574144.709596:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574144.709599:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488735488:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574144.709618:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488735616:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574144.709632:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574144.709640:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574144/real 1788574144] req@ffff94d408d3f800 x1875450488735616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574144.709649:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f800 x1875450488735616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574144.709654:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574144.709656:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574144.709659:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488735616:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574144.709662:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488735744:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574144.709671:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574144.709677:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574144/real 1788574144] req@ffff94d408d3ca80 x1875450488735744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574144.709684:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488735744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574144.709690:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574144.709692:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574144.709693:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488735744:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574144.709697:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488735872:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574144.709707:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574144.709712:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574144/real 1788574144] req@ffff94d408d3d500 x1875450488735872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574144.709719:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488735872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574144.709724:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574144.709726:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574144.709728:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488735872:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574144.709731:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488736000:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574144.709740:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574144.709745:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574144/real 1788574144] req@ffff94d408d3e680 x1875450488736000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574144.709751:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488736000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574144.709756:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574144.709758:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574144.709760:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488736000:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574145.221297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574145.669366:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574146.245372:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574146.693447:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574147.269283:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574147.717364:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574148.293328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574148.741394:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574149.317328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574149.765360:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574149.829274:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574149.829286:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574149.829290:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574149.829302:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5298 00000100:00080000:3.0:1788574149.829315:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574149.829322:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574149.829362:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488736128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574149.829387:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574149.829390:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574149.829392:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574149.829395:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5298 00000100:00080000:3.0:1788574149.829398:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574149.829400:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574149.829416:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574149.829426:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488736256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574149.829434:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488736128:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574149.829437:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574149.829442:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574149.829444:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574149.829447:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5298 00000100:00080000:3.0:1788574149.829450:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574149.829452:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574149.829456:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574149.829468:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488736384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574149.829479:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574149.829484:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574149.829485:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574149.829490:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5298 00000100:00080000:3.0:1788574149.829508:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574149.829513:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574149.829519:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574149.829533:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488736512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574149.829543:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574149.829547:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574149.829549:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574149.829553:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5298 00000100:00080000:3.0:1788574149.829557:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788574149.829560:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488736256:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788574149.829565:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574149.829574:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574149.829590:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488736640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574149.829605:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488736384:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574149.829632:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488736512:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574149.829657:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488736640:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574150.341325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574150.789402:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574151.365311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574151.813374:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574152.389384:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574152.837361:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788574152.901517:0:2392:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:0.0:1788574152.901587:0:2392:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788574152.901610:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:0.0:1788574152.901659:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:0.0:1788574152.901676:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:0.0:1788574152.901689:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:0.0:1788574152.901702:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:2.0:1788574152.901718:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574149/real 1788574152] req@ffff94d408c55f80 x1875450488736256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574152.901754:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488736256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574152.901769:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574152.901775:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574152.901779:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488736256:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574152.901788:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574149/real 1788574152] req@ffff94d408c54000 x1875450488736384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574152.901818:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488736384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574152.901834:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574152.901838:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574152.901841:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488736384:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574152.901849:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574149/real 1788574152] req@ffff94d408c56300 x1875450488736512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574152.901862:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488736512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574152.901871:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574152.901875:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574152.901879:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488736512:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574152.901887:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574149/real 1788574152] req@ffff94d408c54700 x1875450488736128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574152.901902:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488736128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574152.901911:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574152.901915:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574152.901919:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488736128:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574152.901970:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574149/real 1788574152] req@ffff94d408c57480 x1875450488736640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574152.901984:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488736640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574152.901992:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574152.901996:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574152.902000:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488736640:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574153.413316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574153.861455:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574154.437279:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574154.885329:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574154.949273:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574154.949283:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574154.949286:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574154.949294:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5303 00000100:00080000:3.0:1788574154.949304:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574154.949310:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574154.949342:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488736768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574154.949367:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574154.949370:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574154.949371:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574154.949374:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5303 00000100:00080000:3.0:1788574154.949376:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574154.949379:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574154.949386:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574154.949395:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488736896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574154.949403:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574154.949406:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574154.949407:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574154.949410:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5303 00000100:00080000:3.0:1788574154.949412:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574154.949414:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574154.949417:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574154.949421:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488736768:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574154.949427:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488737024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574154.949451:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574154.949455:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574154.949457:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574154.949462:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5303 00000100:00080000:3.0:1788574154.949483:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574154.949489:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:2.0:1788574154.949495:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:3.0:1788574154.949496:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574154.949514:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488737152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574154.949526:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574154.949530:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574154.949533:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574154.949535:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574154/real 1788574154] req@ffff94d408c56a00 x1875450488736768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574154.949540:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5303 00000100:00080000:3.0:1788574154.949548:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574154.949556:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574154.949562:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574154.949566:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488736768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574154.949580:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574154.949586:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574154.949592:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488737280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574154.949593:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488736768:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574154.949609:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488736896:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574154.949638:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574154.949652:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574154/real 1788574154] req@ffff94d408c55880 x1875450488736896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574154.949665:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488736896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574154.949681:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574154.949685:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574154.949689:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488736896:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574154.949716:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488737024:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574154.949737:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574154.949749:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574154/real 1788574154] req@ffff94d408c55c00 x1875450488737024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574154.949763:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488737024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574154.949772:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574154.949776:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574154.949780:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488737024:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574154.949786:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488737152:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574154.949808:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574154.949820:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574154/real 1788574154] req@ffff94d408c57800 x1875450488737152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574154.949832:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488737152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574154.949841:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574154.949845:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574154.949848:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488737152:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574154.949855:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488737280:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574154.949876:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574154.949888:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574154/real 1788574154] req@ffff94d408c55180 x1875450488737280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574154.949901:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488737280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574154.949909:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574154.949913:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574154.949916:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488737280:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574155.461343:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574155.909392:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574156.485280:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574156.933332:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574157.509311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574157.957394:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574158.533334:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574158.981386:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574159.557348:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574160.005424:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574160.069305:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574160.069313:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574160.069317:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574160.069325:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5308 00000100:00080000:3.0:1788574160.069335:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574160.069341:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574160.069375:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488737408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574160.069399:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574160.069402:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574160.069403:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574160.069406:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5308 00000100:00080000:3.0:1788574160.069408:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574160.069410:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574160.069418:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574160.069427:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488737536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574160.069434:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574160.069437:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574160.069438:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574160.069440:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5308 00000100:00080000:3.0:1788574160.069441:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574160.069443:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574160.069446:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574160.069452:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488737664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574160.069458:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574160.069461:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574160.069462:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574160.069466:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5308 00000100:00080000:3.0:1788574160.069467:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574160.069470:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574160.069474:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574160.069481:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488737408:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574160.069482:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488737792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574160.069495:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574160.069497:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574160.069499:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574160.069502:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5308 00000100:00080000:3.0:1788574160.069504:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574160.069506:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574160.069509:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574160.069518:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488737920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574160.069544:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574160.069573:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574160/real 1788574160] req@ffff94d408c57100 x1875450488737408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574160.069586:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488737408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574160.069594:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574160.069596:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574160.069599:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488737408:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574160.069604:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488737536:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574160.069617:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574160.069624:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574160/real 1788574160] req@ffff94d408c54380 x1875450488737536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574160.069632:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488737536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574160.069641:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574160.069643:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574160.069645:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488737536:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574160.069648:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488737664:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574160.069657:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574160.069663:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574160/real 1788574160] req@ffff94d408c54e00 x1875450488737664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574160.069669:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488737664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574160.069674:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574160.069676:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574160.069678:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488737664:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574160.069697:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488737792:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574160.069708:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574160.069713:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574160/real 1788574160] req@ffff94d408c56d80 x1875450488737792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574160.069721:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488737792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574160.069725:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574160.069727:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574160.069729:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488737792:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574160.069733:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488737920:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574160.069742:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574160.069747:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574160/real 1788574160] req@ffff94d408c54a80 x1875450488737920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574160.069754:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488737920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574160.069759:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574160.069761:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574160.069763:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488737920:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574160.581341:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574161.029405:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574161.605497:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574162.053398:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574162.629321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574163.077394:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574163.653615:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574164.101377:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574164.677323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574165.125385:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574165.189312:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574165.189328:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574165.189334:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574165.189347:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5313 00000100:00080000:3.0:1788574165.189363:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574165.189371:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574165.189419:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488738048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574165.189450:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574165.189453:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574165.189455:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574165.189458:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5313 00000100:00080000:3.0:1788574165.189461:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574165.189465:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574165.189475:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574165.189485:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488738176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574165.189494:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574165.189497:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574165.189500:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574165.189507:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488738048:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574165.189507:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5313 00000100:00080000:3.0:1788574165.189510:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574165.189523:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574165.189529:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574165.189542:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488738304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574165.189549:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574165.189552:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574165.189555:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574165.189558:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5313 00000100:00080000:3.0:1788574165.189561:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574165.189571:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574165.189575:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574165.189584:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488738432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574165.189591:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574165.189610:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574165.189613:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574165.189615:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574165.189618:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5313 00000100:00080000:3.0:1788574165.189623:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574165.189641:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574165.189647:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574165.189649:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574165/real 1788574165] req@ffff94d408c56680 x1875450488738048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574165.189668:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488738560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574165.189672:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488738048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574165.189684:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574165.189694:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574165.189703:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488738048:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574165.189728:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488738176:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574165.189748:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574165.189757:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574165/real 1788574165] req@ffff94d408c57b80 x1875450488738176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574165.189765:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488738176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574165.189771:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574165.189773:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574165.189775:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488738176:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574165.189779:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488738304:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574165.189790:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574165.189796:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574165/real 1788574165] req@ffff94d408c54a80 x1875450488738304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574165.189803:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488738304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574165.189808:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574165.189810:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574165.189812:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488738304:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574165.189815:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488738432:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574165.189826:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574165.189832:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574165/real 1788574165] req@ffff94d408c56d80 x1875450488738432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574165.189838:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488738432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574165.189844:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574165.189846:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574165.189848:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488738432:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574165.189851:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488738560:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574165.189861:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574165.189866:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574165/real 1788574165] req@ffff94d408c54e00 x1875450488738560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574165.189873:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488738560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574165.189877:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574165.189879:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574165.189881:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488738560:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574165.701314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574166.149383:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574166.725323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574167.173365:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574167.749267:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574168.197380:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574168.773306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574169.221358:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574169.797302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574170.245393:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574170.309368:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574170.309381:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574170.309385:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574170.309398:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5318 00000100:00080000:1.0:1788574170.309412:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574170.309449:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574170.309500:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488738688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574170.309535:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574170.309539:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574170.309541:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574170.309545:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5318 00000100:00080000:1.0:1788574170.309550:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574170.309555:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574170.309566:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574170.309581:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488738688:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788574170.309584:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488738816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574170.309606:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574170.309612:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574170.309614:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574170.309619:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5318 00000100:00080000:1.0:1788574170.309623:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574170.309626:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574170.309634:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574170.309645:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488738944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574170.309659:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788574170.309661:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574170.309665:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574170.309667:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574170.309673:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5318 00000100:00080000:1.0:1788574170.309677:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574170.309681:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574170.309687:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574170.309708:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488739072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574170.309708:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574170/real 1788574170] req@ffff94d4049bc000 x1875450488738688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574170.309750:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574170.309754:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574170.309757:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574170.309763:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5318 00000100:00100000:2.0:1788574170.309768:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488738688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574170.309770:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574170.309782:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574170.309794:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788574170.309797:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574170.309807:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574170.309812:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488739200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574170.309814:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488738688:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574170.309852:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488738816:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574170.309872:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574170.309881:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574170/real 1788574170] req@ffff94d4049bfb80 x1875450488738816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574170.309889:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488738816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574170.309895:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574170.309897:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574170.309899:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488738816:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574170.309904:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488738944:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574170.309914:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574170.309920:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574170/real 1788574170] req@ffff94d4049bea00 x1875450488738944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574170.309927:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488738944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574170.309933:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574170.309935:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574170.309937:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488738944:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574170.309940:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488739072:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574170.309951:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574170.309957:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574170/real 1788574170] req@ffff94d4049bf480 x1875450488739072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574170.309964:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488739072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574170.309969:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574170.309972:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574170.309973:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488739072:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574170.309977:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488739200:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574170.309989:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574170.309994:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574170/real 1788574170] req@ffff94d4049bc380 x1875450488739200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574170.310011:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc380 x1875450488739200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574170.310016:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574170.310018:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574170.310021:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488739200:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574170.821289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574171.269337:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574171.845357:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574172.293413:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574172.869299:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574173.317348:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574173.893312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574174.341337:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574174.917307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574175.365404:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574175.429391:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574175.429407:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574175.429415:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574175.429431:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5323 00000100:00080000:1.0:1788574175.429449:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574175.429463:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574175.429521:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488739328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574175.429558:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574175.429565:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574175.429568:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574175.429575:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5323 00000100:00080000:1.0:1788574175.429582:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574175.429589:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574175.429603:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574175.429609:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488739328:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788574175.429630:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488739456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574175.429648:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574175.429653:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574175.429656:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574175.429662:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5323 00000100:00080000:1.0:1788574175.429668:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574175.429675:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574175.429684:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788574175.429697:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574175.429706:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488739584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574175.429721:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574175.429726:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574175.429729:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574175.429736:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5323 00000100:00080000:1.0:1788574175.429742:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788574175.429749:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574175/real 1788574175] req@ffff94d4049bce00 x1875450488739328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574175.429755:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574175.429768:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574175.429780:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488739328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574175.429791:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488739712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574175.429847:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574175.429853:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574175.429857:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574175.429858:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574175.429869:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5323 00000100:00080000:2.0:1788574175.429870:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:1.0:1788574175.429879:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788574175.429881:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488739328:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788574175.429890:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574175.429898:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574175.429919:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488739456:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574175.429921:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488739840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574175.429961:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574175.429986:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574175/real 1788574175] req@ffff94d4049bd180 x1875450488739456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574175.430006:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488739456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574175.430019:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574175.430024:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574175.430028:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488739456:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574175.430037:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488739584:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574175.430066:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574175.430084:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574175/real 1788574175] req@ffff94d4049bc380 x1875450488739584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574175.430108:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc380 x1875450488739584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574175.430119:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574175.430139:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574175.430144:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488739584:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574175.430152:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488739712:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574175.430177:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574175.430191:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574175/real 1788574175] req@ffff94d4049bf480 x1875450488739712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574175.430236:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488739712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574175.430246:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574175.430252:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574175.430260:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488739712:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574175.430283:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488739840:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574175.430306:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574175.430318:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574175/real 1788574175] req@ffff94d4049bea00 x1875450488739840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574175.430331:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488739840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574175.430340:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574175.430344:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574175.430348:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488739840:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574175.941316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574176.389363:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574176.965361:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574177.413403:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574177.989281:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574178.437364:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574179.013299:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574179.461383:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574180.037333:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574180.485414:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574180.549366:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574180.549379:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574180.549386:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574180.549399:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5328 00000100:00080000:3.0:1788574180.549414:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574180.549424:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574180.549485:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488739968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574180.549519:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574180.549523:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574180.549526:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574180.549530:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5328 00000100:00080000:3.0:1788574180.549534:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574180.549540:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574180.549553:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574180.549568:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488740096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574180.549581:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488739968:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574180.549586:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574180.549598:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574180.549601:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574180.549606:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5328 00000100:00080000:3.0:1788574180.549611:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574180.549616:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574180.549622:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574180.549638:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488740224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574180.549649:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574180.549654:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574180.549657:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574180.549662:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5328 00000100:00080000:3.0:1788574180.549667:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574180.549674:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:2.0:1788574180.549679:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:3.0:1788574180.549684:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574180.549702:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488740352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574180.549714:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574180.549718:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574180.549721:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574180.549729:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574180/real 1788574180] req@ffff94d408c54380 x1875450488739968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574180.549731:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5328 00000100:00080000:3.0:1788574180.549739:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574180.549751:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574180.549764:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574180.549772:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488739968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574180.549784:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488740480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574180.549790:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574180.549802:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574180.549809:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488739968:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574180.549854:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488740096:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574180.549883:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574180.549896:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574180/real 1788574180] req@ffff94d408c57100 x1875450488740096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574180.549910:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488740096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574180.549918:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574180.549922:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574180.549926:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488740096:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574180.549933:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488740224:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574180.549952:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574180.549963:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574180/real 1788574180] req@ffff94d408c55180 x1875450488740224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574180.549975:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488740224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574180.549990:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574180.549994:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574180.549998:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488740224:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574180.550004:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488740352:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574180.550024:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574180.550034:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574180/real 1788574180] req@ffff94d408c57800 x1875450488740352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574180.550046:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488740352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574180.550054:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574180.550058:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574180.550061:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488740352:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574180.550068:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488740480:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574180.550084:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574180.550094:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574180/real 1788574180] req@ffff94d408c55c00 x1875450488740480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574180.550117:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488740480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574180.550126:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574180.550129:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574180.550132:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488740480:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574181.061350:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574181.509386:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574182.085297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574182.533400:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574183.109306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574183.557463:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574184.133296:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574184.581328:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574185.157290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574185.605365:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574185.669364:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574185.669373:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574185.669377:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574185.669385:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5333 00000100:00080000:2.0:1788574185.669394:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574185.669401:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574185.669434:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488740608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574185.669464:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574185.669467:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574185.669468:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574185.669470:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5333 00000100:00080000:2.0:1788574185.669473:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574185.669476:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574185.669483:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574185.669492:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488740736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574185.669500:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574185.669502:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574185.669504:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574185.669507:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5333 00000100:00080000:2.0:1788574185.669509:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574185.669512:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574185.669515:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574185.669524:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488740608:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574185.669524:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488740864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574185.669549:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574185.669552:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574185.669554:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574185.669556:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5333 00000100:00080000:2.0:1788574185.669559:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574185.669562:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574185.669566:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574185.669576:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488740992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574185.669585:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574185.669589:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574185.669593:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788574185.669594:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788574185.669601:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5333 00000100:00080000:2.0:1788574185.669607:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574185.669612:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574185.669618:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574185.669632:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488741120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574185.669640:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574185/real 1788574185] req@ffff94d408c63b80 x1875450488740608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574185.669664:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488740608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574185.669676:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574185.669680:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574185.669684:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488740608:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574185.669690:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488740736:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574185.669710:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574185.669720:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574185/real 1788574185] req@ffff94d408c61f80 x1875450488740736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574185.669728:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488740736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574185.669734:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574185.669736:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574185.669739:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488740736:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574185.669757:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488740864:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574185.669769:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574185.669775:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574185/real 1788574185] req@ffff94d408c63480 x1875450488740864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574185.669782:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488740864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574185.669788:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574185.669790:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574185.669792:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488740864:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574185.669796:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488740992:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574185.669806:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574185.669812:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574185/real 1788574185] req@ffff94d408c60a80 x1875450488740992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574185.669824:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488740992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574185.669829:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574185.669830:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574185.669833:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488740992:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574185.669838:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488741120:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574185.669856:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574185.669868:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574185/real 1788574185] req@ffff94d408c62a00 x1875450488741120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574185.669881:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488741120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574185.669889:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574185.669892:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574185.669896:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488741120:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574186.181312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574186.629328:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574187.205299:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574187.653334:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574188.229288:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574188.677329:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574189.253271:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574189.701374:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574190.277274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574190.725391:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574190.789336:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574190.789347:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574190.789352:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574190.789362:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5339 00000100:00080000:3.0:1788574190.789374:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574190.789382:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574190.789423:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488741248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574190.789454:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574190.789458:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574190.789460:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574190.789464:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5339 00000100:00080000:3.0:1788574190.789468:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574190.789472:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574190.789480:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574190.789492:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488741376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574190.789502:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788574190.789512:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488741248:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574190.789514:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574190.789518:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574190.789525:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5339 00000100:00080000:3.0:1788574190.789527:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574190.789529:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574190.789532:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574190.789542:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488741504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574190.789549:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574190.789552:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574190.789553:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574190.789555:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5339 00000100:00080000:3.0:1788574190.789558:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574190.789560:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574190.789564:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574190.789570:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488741632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574190.789577:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574190.789579:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574190.789580:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788574190.789585:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574190.789589:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5339 00000100:00080000:3.0:1788574190.789593:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574190.789595:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574190.789599:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574190.789607:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488741760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574190.789631:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574190/real 1788574190] req@ffff94d408c55880 x1875450488741248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574190.789650:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488741248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574190.789667:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574190.789671:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574190.789676:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488741248:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574190.789697:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488741376:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574190.789719:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574190.789731:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574190/real 1788574190] req@ffff94d408c56a00 x1875450488741376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574190.789744:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488741376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574190.789752:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574190.789755:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574190.789758:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488741376:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574190.789764:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488741504:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574190.789781:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574190.789790:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574190/real 1788574190] req@ffff94d408c57480 x1875450488741504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574190.789802:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488741504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574190.789811:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574190.789823:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574190.789828:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488741504:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574190.789834:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488741632:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574190.789855:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574190.789867:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574190/real 1788574190] req@ffff94d408c54700 x1875450488741632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574190.789879:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488741632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574190.789887:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574190.789890:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574190.789894:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488741632:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574190.789899:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488741760:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574190.789916:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574190.789926:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574190/real 1788574190] req@ffff94d408c56300 x1875450488741760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574190.789937:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488741760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574190.789945:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574190.789948:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574190.789951:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488741760:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574191.301289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574191.749334:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574192.325285:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574192.773335:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574193.349295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574193.797325:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574194.373282:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574194.821426:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574195.397279:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574195.845361:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574195.909329:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574195.909339:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574195.909344:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574195.909355:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5344 00000100:00080000:3.0:1788574195.909368:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574195.909376:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574195.909428:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488741888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574195.909455:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574195.909458:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574195.909460:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574195.909463:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5344 00000100:00080000:3.0:1788574195.909465:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574195.909468:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574195.909478:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574195.909487:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488742016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574195.909498:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574195.909501:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574195.909502:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574195.909505:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5344 00000100:00080000:3.0:1788574195.909507:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574195.909509:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574195.909512:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574195.909521:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488742144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574195.909527:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574195.909530:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574195.909532:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574195.909535:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5344 00000100:00080000:3.0:1788574195.909538:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574195.909540:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574195.909544:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574195.909546:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488741888:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574195.909554:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488742272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574195.909561:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574195.909563:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574195.909565:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574195.909567:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5344 00000100:00080000:3.0:1788574195.909570:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574195.909572:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574195.909575:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574195.909586:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488742400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574195.909625:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574195.909673:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574195/real 1788574195] req@ffff94d408c54000 x1875450488741888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574195.909688:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488741888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574195.909698:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574195.909701:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574195.909705:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488741888:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574195.909710:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488742016:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574195.909726:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574195.909735:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574195/real 1788574195] req@ffff94d408c55f80 x1875450488742016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574195.909743:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488742016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574195.909750:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574195.909752:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574195.909756:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488742016:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574195.909762:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488742144:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574195.909783:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574195.909795:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574195/real 1788574195] req@ffff94d408c56300 x1875450488742144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574195.909808:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488742144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574195.909823:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574195.909832:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574195.909835:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488742144:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574195.909860:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488742272:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574195.909877:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574195.909885:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574195/real 1788574195] req@ffff94d408c54700 x1875450488742272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574195.909893:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488742272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574195.909898:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574195.909900:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574195.909902:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488742272:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574195.909906:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488742400:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574195.909915:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574195.909920:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574195/real 1788574195] req@ffff94d408c57480 x1875450488742400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574195.909927:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488742400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574195.909933:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574195.909934:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574195.909952:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488742400:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574196.421350:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574196.869467:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574197.445316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574197.893374:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574198.469350:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574198.917412:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574199.493294:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574199.941339:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574200.517287:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574200.965392:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574201.029304:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574201.029315:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574201.029320:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574201.029330:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5349 00000100:00080000:3.0:1788574201.029340:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574201.029348:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574201.029384:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488742528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574201.029417:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574201.029421:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574201.029423:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574201.029427:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5349 00000100:00080000:3.0:1788574201.029431:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574201.029436:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574201.029445:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574201.029459:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488742656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574201.029463:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488742528:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574201.029481:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574201.029485:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574201.029487:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574201.029493:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5349 00000100:00080000:3.0:1788574201.029497:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574201.029501:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574201.029507:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574201.029520:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488742784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574201.029528:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574201.029537:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574201.029540:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574201.029543:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574201.029548:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5349 00000100:00080000:3.0:1788574201.029553:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574201.029557:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574201.029563:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574201.029575:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574201/real 1788574201] req@ffff94d408c56a00 x1875450488742528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574201.029577:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488742912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574201.029605:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488742528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574201.029611:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574201.029618:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574201.029619:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574201.029621:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574201.029625:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:3.0:1788574201.029629:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5349 00000100:00100000:0.0:1788574201.029631:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488742528:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574201.029637:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574201.029659:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574201.029666:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574201.029672:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488742656:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574201.029681:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488743040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574201.029696:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574201.029708:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574201/real 1788574201] req@ffff94d408c55880 x1875450488742656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574201.029718:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488742656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574201.029724:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574201.029726:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574201.029728:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488742656:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574201.029741:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488742784:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574201.029753:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574201.029761:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574201/real 1788574201] req@ffff94d408c55c00 x1875450488742784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574201.029769:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488742784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574201.029774:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574201.029776:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574201.029778:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488742784:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574201.029782:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488742912:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574201.029798:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574201.029805:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574201/real 1788574201] req@ffff94d408c57800 x1875450488742912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574201.029813:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488742912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574201.029818:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574201.029820:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574201.029822:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488742912:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574201.029834:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488743040:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574201.029854:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574201.029867:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574201/real 1788574201] req@ffff94d408c55180 x1875450488743040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574201.029879:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488743040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574201.029886:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574201.029888:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574201.029890:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488743040:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574201.541338:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574201.989405:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574202.565338:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574203.013445:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574203.589315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574204.037373:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574204.613321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574205.061356:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574205.637308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574206.085369:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574206.149275:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574206.149283:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574206.149286:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574206.149294:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5354 00000100:00080000:3.0:1788574206.149304:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574206.149312:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574206.149373:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488743168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574206.149396:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574206.149399:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574206.149401:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574206.149403:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5354 00000100:00080000:3.0:1788574206.149406:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574206.149410:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574206.149417:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574206.149429:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488743296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574206.149438:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488743168:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574206.149442:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574206.149447:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574206.149449:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574206.149455:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5354 00000100:00080000:3.0:1788574206.149459:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574206.149464:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574206.149470:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574206.149485:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488743424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574206.149494:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788574206.149498:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574206.149500:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574206.149503:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574206.149507:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5354 00000100:00080000:3.0:1788574206.149511:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574206.149516:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574206.149521:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574206.149530:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488743552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574206.149534:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574206/real 1788574206] req@ffff94d408c57100 x1875450488743168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574206.149544:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574206.149552:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574206.149555:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574206.149559:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5354 00000100:00080000:3.0:1788574206.149563:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788574206.149564:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488743168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574206.149572:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00080000:0.0:1788574206.149578:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788574206.149580:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788574206.149584:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574206.149590:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488743168:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574206.149599:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488743680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574206.149611:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488743296:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574206.149631:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574206.149640:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574206/real 1788574206] req@ffff94d408c54380 x1875450488743296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574206.149649:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488743296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574206.149655:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574206.149657:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574206.149659:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488743296:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574206.149663:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488743424:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574206.149674:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574206.149680:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574206/real 1788574206] req@ffff94d408c54e00 x1875450488743424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574206.149687:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488743424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574206.149693:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574206.149694:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574206.149696:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488743424:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574206.149700:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488743552:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574206.149710:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574206.149715:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574206/real 1788574206] req@ffff94d408c56d80 x1875450488743552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574206.149722:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488743552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574206.149727:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574206.149729:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574206.149731:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488743552:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574206.149740:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488743680:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574206.149763:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574206.149771:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574206/real 1788574206] req@ffff94d408c54a80 x1875450488743680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574206.149779:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488743680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574206.149784:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574206.149786:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574206.149788:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488743680:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574206.661329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574207.109354:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574207.685359:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574208.133405:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574208.709306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574209.157359:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574209.733286:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574210.181362:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574210.757309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574211.205331:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574211.269370:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574211.269386:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574211.269391:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574211.269406:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5359 00000100:00080000:3.0:1788574211.269421:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574211.269431:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574211.269483:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488743808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574211.269526:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574211.269531:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574211.269533:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574211.269538:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5359 00000100:00080000:3.0:1788574211.269542:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574211.269548:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574211.269561:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574211.269578:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488743808:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574211.269581:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488743936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574211.269598:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574211.269603:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574211.269606:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574211.269611:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5359 00000100:00080000:3.0:1788574211.269617:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574211.269622:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574211.269629:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574211.269644:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488744064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574211.269656:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574211.269657:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574211.269663:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574211.269666:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574211.269674:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5359 00000100:00080000:3.0:1788574211.269679:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574211.269685:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574211.269692:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574211.269694:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574211/real 1788574211] req@ffff94d408c57b80 x1875450488743808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574211.269712:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488743808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574211.269722:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574211.269727:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574211.269731:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488743808:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574211.269759:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488743936:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574211.269779:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574211.269798:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574211/real 1788574211] req@ffff94d408c56680 x1875450488743936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574211.269817:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488743936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574211.269827:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574211.269832:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574211.269836:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488743936:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574211.269842:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488744064:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574211.269859:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574211.269868:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574211/real 1788574211] req@ffff94d408c54a80 x1875450488744064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574211.269883:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488744064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574211.269889:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574211.269891:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574211.269894:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488744064:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574211.269945:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488744192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574211.269990:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488744192:192.168.202.125@tcp:8:kworker.0 00000100:00080000:3.0:1788574211.270009:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574211.270015:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1788574211.270018:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574211.270020:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574211.270028:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5359 00000100:00100000:2.0:1788574211.270033:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574211/real 1788574211] req@ffff94d408c56d80 x1875450488744192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574211.270036:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574211.270046:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574211.270061:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488744192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788574211.270072:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788574211.270073:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574211.270079:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574211.270083:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488744192:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574211.270094:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488744320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574211.270126:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488744320:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574211.270145:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574211.270153:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574211/real 1788574211] req@ffff94d408c54e00 x1875450488744320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574211.270161:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488744320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574211.270172:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574211.270174:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574211.270177:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488744320:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574211.781295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574212.229439:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574212.805317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574213.253406:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574213.829305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574214.277460:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574214.853287:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574215.301382:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574215.877253:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574216.325326:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574216.389325:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574216.389334:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574216.389338:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574216.389347:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5364 00000100:00080000:3.0:1788574216.389356:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574216.389364:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574216.389405:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488744448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574216.389431:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574216.389433:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574216.389435:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574216.389438:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5364 00000100:00080000:3.0:1788574216.389440:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574216.389444:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574216.389453:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574216.389465:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488744576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574216.389467:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488744448:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574216.389481:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574216.389484:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574216.389485:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574216.389488:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5364 00000100:00080000:3.0:1788574216.389491:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574216.389493:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574216.389496:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574216.389505:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488744704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574216.389515:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574216.389518:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574216.389519:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574216.389522:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5364 00000100:00080000:3.0:1788574216.389524:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574216.389526:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574216.389529:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574216.389536:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488744832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574216.389543:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574216.389546:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574216.389547:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574216.389551:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5364 00000100:00080000:3.0:1788574216.389553:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574216.389556:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574216.389559:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574216.389561:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488744576:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574216.389569:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488744960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574216.389594:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488744704:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574216.389611:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488744832:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574216.389625:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488744960:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574216.901263:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574217.349339:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574217.925259:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574218.373439:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574218.949286:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574219.397508:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788574219.461478:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:1.0:1788574219.461520:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788574219.461535:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788574219.461591:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788574219.461602:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788574219.461613:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788574219.461623:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:2.0:1788574219.461649:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574216/real 1788574219] req@ffff94d408c55c00 x1875450488744960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574219.461679:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488744960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574219.461692:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574219.461697:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574219.461702:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488744960:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574219.461711:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574216/real 1788574219] req@ffff94d408c57100 x1875450488744576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574219.461726:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488744576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574219.461742:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574219.461746:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574219.461750:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488744576:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574219.461758:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574216/real 1788574219] req@ffff94d408c55180 x1875450488744704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574219.461772:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488744704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574219.461787:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574219.461791:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574219.461795:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488744704:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574219.461802:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574216/real 1788574219] req@ffff94d408c57800 x1875450488744832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574219.461813:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488744832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574219.461822:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574219.461826:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574219.461830:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488744832:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574219.461837:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574216/real 1788574219] req@ffff94d408c54380 x1875450488744448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574219.461853:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488744448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574219.461862:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574219.461866:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574219.461869:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488744448:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574219.973278:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574220.421388:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574220.997298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574221.445356:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574221.509306:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574221.509314:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574221.509317:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574221.509327:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5369 00000100:00080000:3.0:1788574221.509336:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574221.509342:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574221.509371:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488745088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574221.509393:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574221.509396:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574221.509397:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574221.509401:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5369 00000100:00080000:3.0:1788574221.509403:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574221.509405:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574221.509412:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574221.509421:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488745216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574221.509429:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574221.509432:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574221.509434:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574221.509436:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5369 00000100:00080000:3.0:1788574221.509438:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574221.509441:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574221.509445:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574221.509455:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488745344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574221.509456:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488745088:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574221.509483:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574221.509485:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574221.509487:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574221.509490:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5369 00000100:00080000:3.0:1788574221.509492:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574221.509495:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574221.509501:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574221.509513:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488745472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574221.509523:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574221.509527:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574221.509529:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574221.509533:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5369 00000100:00080000:3.0:1788574221.509545:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788574221.509554:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574221.509554:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574221.509563:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574221.509578:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488745600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574221.509608:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574221/real 1788574221] req@ffff94d408c55880 x1875450488745088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574221.509629:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488745088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574221.509640:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574221.509644:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574221.509649:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488745088:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574221.509657:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488745216:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574221.509681:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574221.509693:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574221/real 1788574221] req@ffff94d408c56a00 x1875450488745216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574221.509705:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488745216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574221.509714:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574221.509718:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574221.509721:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488745216:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574221.509751:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488745344:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574221.509771:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574221.509782:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574221/real 1788574221] req@ffff94d408c57480 x1875450488745344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574221.509793:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488745344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574221.509802:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574221.509806:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574221.509809:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488745344:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574221.509815:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488745472:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574221.509835:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574221.509853:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574221/real 1788574221] req@ffff94d408c54700 x1875450488745472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574221.509867:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488745472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574221.509875:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574221.509879:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574221.509882:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488745472:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574221.509888:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488745600:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574221.509908:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574221.509919:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574221/real 1788574221] req@ffff94d408c56300 x1875450488745600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574221.509931:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488745600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574221.509938:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574221.509941:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574221.509945:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488745600:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574222.021326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574222.469430:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574223.045278:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574223.493427:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574224.069347:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574224.517387:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574225.093399:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574225.541376:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574226.117329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574226.565338:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574226.629300:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574226.629308:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574226.629312:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574226.629321:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5374 00000100:00080000:2.0:1788574226.629332:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574226.629341:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574226.629378:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488745728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574226.629404:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574226.629407:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574226.629408:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574226.629411:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5374 00000100:00080000:2.0:1788574226.629415:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574226.629418:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574226.629425:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574226.629435:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488745856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574226.629447:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574226.629450:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574226.629451:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574226.629456:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488745728:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788574226.629457:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5374 00000100:00080000:2.0:1788574226.629463:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574226.629467:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574226.629469:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574226.629477:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488745984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574226.629483:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574226.629486:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574226.629487:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574226.629490:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5374 00000100:00080000:2.0:1788574226.629492:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574226.629495:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574226.629498:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574226.629505:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488746112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574226.629511:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574226.629513:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574226.629514:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788574226.629519:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788574226.629521:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5374 00000100:00080000:2.0:1788574226.629526:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574226.629528:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574226.629532:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574226.629540:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488746240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574226.629561:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574226/real 1788574226] req@ffff94d408c62d80 x1875450488745728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574226.629581:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488745728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574226.629591:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574226.629595:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574226.629598:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488745728:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574226.629621:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488745856:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574226.629645:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574226.629658:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574226/real 1788574226] req@ffff94d408c60000 x1875450488745856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574226.629670:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488745856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574226.629683:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574226.629686:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574226.629689:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488745856:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574226.629695:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488745984:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574226.629714:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574226.629737:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574226/real 1788574226] req@ffff94d408c63800 x1875450488745984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574226.629750:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488745984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574226.629758:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574226.629760:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574226.629762:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488745984:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574226.629767:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488746112:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574226.629782:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574226.629790:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574226/real 1788574226] req@ffff94d408c62680 x1875450488746112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574226.629797:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488746112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574226.629802:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574226.629804:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574226.629806:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488746112:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574226.629810:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488746240:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574226.629821:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574226.629826:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574226/real 1788574226] req@ffff94d408c61180 x1875450488746240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574226.629833:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488746240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574226.629838:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574226.629840:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574226.629841:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488746240:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574227.141314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574227.589317:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574228.165327:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574228.613331:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574229.189629:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574229.637323:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574230.213375:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574230.661354:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574231.237261:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574231.685344:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574231.749310:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574231.749321:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574231.749326:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574231.749336:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5380 00000100:00080000:2.0:1788574231.749347:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574231.749354:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574231.749401:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60700 x1875450488746368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574231.749432:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574231.749436:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574231.749438:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574231.749442:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5380 00000100:00080000:2.0:1788574231.749446:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574231.749450:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574231.749460:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574231.749472:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60380 x1875450488746496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574231.749482:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574231.749489:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574231.749491:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574231.749495:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5380 00000100:00080000:2.0:1788574231.749499:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788574231.749504:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488746368:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788574231.749506:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574231.749516:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574231.749530:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62300 x1875450488746624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574231.749541:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574231.749545:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574231.749547:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574231.749551:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5380 00000100:00080000:2.0:1788574231.749555:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574231.749560:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574231.749565:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788574231.749572:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574231.749580:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488746752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574231.749592:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574231.749595:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574231.749598:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574231.749602:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5380 00000100:00080000:2.0:1788574231.749607:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788574231.749608:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574231/real 1788574231] req@ffff94d408c60700 x1875450488746368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574231.749615:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574231.749625:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574231.749631:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60700 x1875450488746368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574231.749640:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488746880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574231.749642:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574231.749650:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574231.749657:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488746368:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574231.749667:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488746496:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574231.749685:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574231.749694:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574231/real 1788574231] req@ffff94d408c60380 x1875450488746496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574231.749702:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60380 x1875450488746496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574231.749708:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574231.749710:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574231.749713:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488746496:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574231.749731:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488746624:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574231.749742:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574231.749748:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574231/real 1788574231] req@ffff94d408c62300 x1875450488746624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574231.749756:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62300 x1875450488746624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574231.749761:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574231.749763:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574231.749765:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488746624:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574231.749769:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488746752:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574231.749779:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574231.749784:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574231/real 1788574231] req@ffff94d408c61180 x1875450488746752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574231.749792:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488746752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574231.749797:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574231.749799:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574231.749801:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488746752:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574231.749804:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488746880:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574231.749814:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574231.749820:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574231/real 1788574231] req@ffff94d408c62680 x1875450488746880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574231.749827:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488746880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574231.749832:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574231.749834:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574231.749836:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488746880:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574232.261291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574232.709346:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574233.285290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574233.733441:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574234.309296:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574234.757334:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574235.333284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574235.781378:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574236.357312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574236.805356:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574236.869334:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574236.869343:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574236.869346:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574236.869354:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5385 00000100:00080000:0.0:1788574236.869364:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574236.869370:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574236.869402:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488747008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574236.869427:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574236.869430:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574236.869432:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574236.869434:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5385 00000100:00080000:0.0:1788574236.869437:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574236.869440:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574236.869447:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574236.869456:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488747136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574236.869461:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488747008:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788574236.869468:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574236.869471:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574236.869472:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574236.869475:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5385 00000100:00080000:0.0:1788574236.869478:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574236.869481:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574236.869484:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574236.869491:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488747264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574236.869497:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574236.869499:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574236.869501:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574236.869504:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5385 00000100:00080000:0.0:1788574236.869506:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574236.869509:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574236.869512:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574236.869519:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488747392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574236.869524:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788574236.869533:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574236.869536:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574236.869537:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574236.869541:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5385 00000100:00080000:0.0:1788574236.869544:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574236.869546:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574236.869549:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574236.869556:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488747520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574236.869561:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574236/real 1788574236] req@ffff94d408d3ea00 x1875450488747008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574236.869578:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488747008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574236.869587:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574236.869590:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574236.869594:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488747008:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574236.869611:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488747136:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574236.869626:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574236.869634:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574236/real 1788574236] req@ffff94d408d3ce00 x1875450488747136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574236.869642:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488747136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574236.869648:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574236.869650:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574236.869660:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488747136:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574236.869664:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488747264:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574236.869678:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574236.869685:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574236/real 1788574236] req@ffff94d408d3df80 x1875450488747264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574236.869693:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488747264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574236.869698:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574236.869701:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574236.869702:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488747264:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574236.869706:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488747392:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574236.869743:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574236.869752:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574236/real 1788574236] req@ffff94d408d3d180 x1875450488747392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574236.869760:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488747392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574236.869769:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574236.869772:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574236.869776:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488747392:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574236.869782:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488747520:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574236.869811:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574236.869824:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574236/real 1788574236] req@ffff94d408d3c000 x1875450488747520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574236.869837:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488747520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574236.869842:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574236.869844:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574236.869847:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488747520:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574237.381274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574237.829368:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574238.405262:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574238.853361:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574239.429300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574239.877392:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574240.453287:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574240.901390:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574241.477296:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574241.925371:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574241.989274:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574241.989282:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574241.989286:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574241.989294:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5390 00000100:00080000:0.0:1788574241.989309:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574241.989316:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574241.989348:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488747648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574241.989374:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574241.989377:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574241.989379:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574241.989381:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5390 00000100:00080000:0.0:1788574241.989384:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574241.989386:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574241.989393:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574241.989401:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488747776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574241.989408:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574241.989411:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574241.989412:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574241.989414:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5390 00000100:00080000:0.0:1788574241.989416:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574241.989418:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574241.989421:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574241.989428:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488747904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574241.989435:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574241.989438:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574241.989439:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574241.989443:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5390 00000100:00080000:0.0:1788574241.989445:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788574241.989445:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488747648:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788574241.989451:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574241.989458:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574241.989465:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488748032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574241.989476:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574241.989478:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574241.989479:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574241.989482:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5390 00000100:00080000:0.0:1788574241.989484:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574241.989486:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574241.989489:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574241.989496:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488748160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574241.989517:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574241.989553:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574241/real 1788574241] req@ffff94d408d3c380 x1875450488747648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574241.989573:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488747648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574241.989585:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574241.989589:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574241.989594:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488747648:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574241.989602:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488747776:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574241.989623:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574241.989649:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574241/real 1788574241] req@ffff94d408d3f480 x1875450488747776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574241.989665:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488747776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574241.989681:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574241.989685:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574241.989689:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488747776:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574241.989713:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488747904:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574241.989734:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574241.989745:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574241/real 1788574241] req@ffff94d408d3c700 x1875450488747904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574241.989758:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488747904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574241.989768:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574241.989771:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574241.989775:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488747904:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574241.989782:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488748032:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574241.989800:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574241.989810:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574241/real 1788574241] req@ffff94d408d3e300 x1875450488748032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574241.989824:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488748032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574241.989832:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574241.989836:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574241.989840:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488748032:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574241.989846:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488748160:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574241.989864:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574241.989874:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574241/real 1788574241] req@ffff94d408d3dc00 x1875450488748160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574241.989886:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488748160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574241.989895:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574241.989899:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574241.989902:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488748160:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574242.501351:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574242.949333:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574243.525327:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574243.973336:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574244.549299:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574244.997354:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574245.573295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574246.021385:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574246.597302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574247.045364:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574247.109336:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574247.109351:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574247.109358:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574247.109373:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5395 00000100:00080000:2.0:1788574247.109389:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574247.109400:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574247.109451:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488748288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574247.109488:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574247.109494:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574247.109497:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574247.109502:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5395 00000100:00080000:2.0:1788574247.109507:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574247.109514:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574247.109527:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574247.109534:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488748288:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574247.109549:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488748416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574247.109581:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574247.109586:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574247.109589:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574247.109594:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5395 00000100:00080000:2.0:1788574247.109600:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574247.109616:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574247.109623:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574247.109642:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488748544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574247.109643:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788574247.109657:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574247.109661:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574247.109664:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574247.109669:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5395 00000100:00080000:2.0:1788574247.109681:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574247.109687:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574247.109698:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574247/real 1788574247] req@ffff94d408c63800 x1875450488748288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788574247.109704:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574247.109725:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488748672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574247.109732:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488748288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574247.109742:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574247.109751:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574247.109754:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574247.109759:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5395 00000100:00080000:0.0:1788574247.109762:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574247.109764:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574247.109774:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00080000:0.0:1788574247.109776:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000008:00100000:2.0:1788574247.109784:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574247.109785:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488748288:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574247.109805:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488748800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574247.109823:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488748416:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574247.109855:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574247.109872:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574247/real 1788574247] req@ffff94d408c60000 x1875450488748416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574247.109887:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488748416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574247.109897:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574247.109901:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574247.109905:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488748416:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574247.109912:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488748544:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574247.109938:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574247.109952:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574247/real 1788574247] req@ffff94d408c62d80 x1875450488748544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574247.109967:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488748544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574247.109976:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574247.109980:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574247.109985:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488748544:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574247.109994:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488748672:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574247.110021:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574247.110034:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574247/real 1788574247] req@ffff94d408c62a00 x1875450488748672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574247.110050:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488748672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574247.110066:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574247.110071:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574247.110076:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488748672:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574247.110106:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488748800:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574247.110132:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574247.110145:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574247/real 1788574247] req@ffff94d408c60a80 x1875450488748800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574247.110157:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488748800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574247.110165:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574247.110169:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574247.110172:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488748800:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574247.621314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574248.069403:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574248.645298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574249.093379:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574249.669303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574250.117382:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574250.693275:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574251.141338:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574251.717324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574252.165370:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574252.229325:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574252.229333:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574252.229337:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574252.229345:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5400 00000100:00080000:2.0:1788574252.229355:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574252.229363:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574252.229403:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488748928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574252.229436:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574252.229440:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574252.229442:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574252.229446:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5400 00000100:00080000:2.0:1788574252.229450:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574252.229455:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574252.229464:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574252.229491:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488749056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574252.229502:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488748928:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788574252.229506:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574252.229511:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574252.229512:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574252.229517:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5400 00000100:00080000:2.0:1788574252.229520:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574252.229523:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574252.229528:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574252.229541:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488749184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574252.229555:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574252.229557:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574252.229559:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788574252.229563:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788574252.229565:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5400 00000100:00080000:2.0:1788574252.229570:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574252.229572:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574252.229577:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574252.229585:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488749312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574252.229592:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574252.229595:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574252.229597:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574252.229598:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574252/real 1788574252] req@ffff94d408c63480 x1875450488748928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574252.229601:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5400 00000100:00080000:2.0:1788574252.229607:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574252.229613:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574252.229618:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574252.229625:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488748928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574252.229627:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488749440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574252.229653:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574252.229659:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574252.229663:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488748928:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574252.229685:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488749056:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574252.229700:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574252.229708:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574252/real 1788574252] req@ffff94d408c61f80 x1875450488749056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574252.229716:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488749056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574252.229721:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574252.229724:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574252.229726:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488749056:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574252.229729:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488749184:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574252.229739:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574252.229745:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574252/real 1788574252] req@ffff94d408c63b80 x1875450488749184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574252.229752:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488749184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574252.229758:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574252.229759:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574252.229761:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488749184:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574252.229764:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488749312:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574252.229774:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574252.229780:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574252/real 1788574252] req@ffff94d408c61500 x1875450488749312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574252.229787:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488749312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574252.229792:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574252.229794:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574252.229796:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488749312:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574252.229799:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488749440:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574252.229808:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574252.229813:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574252/real 1788574252] req@ffff94d408c61c00 x1875450488749440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574252.229821:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488749440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574252.229825:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574252.229827:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574252.229829:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488749440:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574252.741274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574253.189358:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574253.765291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574254.213379:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574254.789296:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574255.237380:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574255.813274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574256.261342:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574256.837314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574257.285361:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574257.349308:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574257.349317:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574257.349320:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574257.349328:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5405 00000100:00080000:3.0:1788574257.349338:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574257.349344:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574257.349378:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488749568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574257.349412:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574257.349415:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574257.349417:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574257.349420:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5405 00000100:00080000:3.0:1788574257.349423:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574257.349427:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574257.349435:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574257.349450:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488749568:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574257.349450:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488749696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574257.349465:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574257.349467:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574257.349469:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574257.349472:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5405 00000100:00080000:3.0:1788574257.349474:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574257.349477:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574257.349480:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574257.349490:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488749824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574257.349495:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574257.349498:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574257.349499:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574257.349502:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5405 00000100:00080000:3.0:1788574257.349505:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574257.349507:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574257.349511:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574257.349518:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488749952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574257.349525:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574257.349527:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574257.349532:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574257.349535:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574257.349541:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5405 00000100:00080000:3.0:1788574257.349548:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574257.349552:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574257.349558:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574257.349562:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574257/real 1788574257] req@ffff94d408c55f80 x1875450488749568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574257.349573:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488750080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574257.349580:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488749568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574257.349606:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574257.349609:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574257.349612:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488749568:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574257.349632:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488749696:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574257.349649:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574257.349658:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574257/real 1788574257] req@ffff94d408c54000 x1875450488749696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574257.349666:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488749696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574257.349671:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574257.349674:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574257.349676:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488749696:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574257.349679:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488749824:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574257.349689:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574257.349699:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574257/real 1788574257] req@ffff94d408c56300 x1875450488749824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574257.349712:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488749824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574257.349721:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574257.349724:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574257.349728:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488749824:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574257.349735:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488749952:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574257.349755:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574257.349764:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574257/real 1788574257] req@ffff94d408c54700 x1875450488749952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574257.349772:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488749952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574257.349778:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574257.349780:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574257.349782:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488749952:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574257.349786:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488750080:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574257.349797:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574257.349803:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574257/real 1788574257] req@ffff94d408c57480 x1875450488750080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574257.349810:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488750080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574257.349815:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574257.349816:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574257.349818:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488750080:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574257.861277:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574258.309337:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574258.885308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574259.333314:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574259.909293:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574260.357365:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574260.933318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574261.381390:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574261.957355:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574262.405340:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574262.469328:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574262.469338:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574262.469343:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574262.469354:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5410 00000100:00080000:3.0:1788574262.469365:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574262.469372:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574262.469403:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488750208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574262.469427:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574262.469429:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574262.469431:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574262.469434:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5410 00000100:00080000:3.0:1788574262.469436:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574262.469439:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574262.469446:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574262.469455:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488750336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574262.469462:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574262.469465:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574262.469466:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574262.469469:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5410 00000100:00080000:3.0:1788574262.469471:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574262.469473:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574262.469477:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574262.469479:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488750208:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574262.469486:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488750464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574262.469494:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574262.469496:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574262.469497:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574262.469500:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5410 00000100:00080000:3.0:1788574262.469503:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574262.469505:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574262.469508:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574262.469516:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488750592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574262.469523:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574262.469526:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574262.469527:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574262.469529:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5410 00000100:00080000:3.0:1788574262.469534:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574262.469536:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574262.469539:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788574262.469545:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574262.469549:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488750720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574262.469581:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574262/real 1788574262] req@ffff94d408c56a00 x1875450488750208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574262.469594:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488750208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574262.469601:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574262.469604:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574262.469607:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488750208:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574262.469612:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488750336:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574262.469626:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574262.469634:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574262/real 1788574262] req@ffff94d408c55880 x1875450488750336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574262.469642:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488750336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574262.469647:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574262.469649:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574262.469651:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488750336:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574262.469669:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488750464:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574262.469680:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574262.469686:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574262/real 1788574262] req@ffff94d408c54380 x1875450488750464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574262.469694:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488750464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574262.469703:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574262.469705:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574262.469707:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488750464:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574262.469711:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488750592:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574262.469721:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574262.469726:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574262/real 1788574262] req@ffff94d408c57800 x1875450488750592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574262.469733:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488750592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574262.469738:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574262.469740:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574262.469742:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488750592:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574262.469745:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488750720:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574262.469754:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574262.469759:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574262/real 1788574262] req@ffff94d408c55180 x1875450488750720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574262.469766:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488750720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574262.469771:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574262.469772:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574262.469775:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488750720:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574262.981336:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574263.429421:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574264.005361:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574264.453423:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574265.029316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574265.477395:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574266.053355:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574266.501388:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574267.077320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574267.525381:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574267.589316:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574267.589327:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574267.589331:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574267.589342:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5415 00000100:00080000:3.0:1788574267.589353:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574267.589361:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574267.589398:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488750848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574267.589429:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574267.589432:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574267.589434:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574267.589437:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5415 00000100:00080000:3.0:1788574267.589440:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574267.589444:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574267.589451:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574267.589464:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488750976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574267.589473:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574267.589475:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574267.589477:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574267.589481:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488750848:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574267.589482:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5415 00000100:00080000:3.0:1788574267.589488:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574267.589491:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574267.589494:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574267.589504:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488751104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574267.589510:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574267.589513:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574267.589514:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574267.589516:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5415 00000100:00080000:3.0:1788574267.589519:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574267.589522:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574267.589525:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574267.589532:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488751232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574267.589538:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788574267.589544:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574267.589544:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574267.589549:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574267.589552:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5415 00000100:00080000:3.0:1788574267.589555:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574267.589557:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574267.589560:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574267.589568:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488751360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574267.589582:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574267/real 1788574267] req@ffff94d408c57100 x1875450488750848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574267.589605:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488750848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574267.589612:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574267.589615:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574267.589618:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488750848:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574267.589637:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488750976:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574267.589652:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574267.589661:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574267/real 1788574267] req@ffff94d408c55c00 x1875450488750976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574267.589669:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488750976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574267.589674:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574267.589676:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574267.589678:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488750976:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574267.589682:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488751104:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574267.589691:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574267.589698:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574267/real 1788574267] req@ffff94d408c54e00 x1875450488751104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574267.589705:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488751104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574267.589711:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574267.589712:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574267.589714:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488751104:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574267.589718:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488751232:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574267.589728:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574267.589747:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574267/real 1788574267] req@ffff94d408c56d80 x1875450488751232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574267.589760:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488751232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574267.589769:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574267.589772:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574267.589776:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488751232:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574267.589782:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488751360:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574267.589800:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574267.589809:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574267/real 1788574267] req@ffff94d408c54a80 x1875450488751360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574267.589825:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488751360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574267.589833:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574267.589836:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574267.589840:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488751360:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574268.101289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574268.549383:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574269.125330:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574269.573341:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574270.149290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574270.597349:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574271.173294:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574271.621360:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574272.197283:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574272.645372:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574272.709320:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574272.709328:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574272.709331:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574272.709340:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5421 00000100:00080000:3.0:1788574272.709350:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574272.709356:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574272.709390:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488751488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574272.709411:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574272.709414:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574272.709416:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574272.709418:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5421 00000100:00080000:3.0:1788574272.709420:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574272.709423:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574272.709431:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574272.709438:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488751616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574272.709446:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574272.709449:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574272.709450:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574272.709453:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5421 00000100:00080000:3.0:1788574272.709455:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574272.709459:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574272.709465:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574272.709472:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488751744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574272.709479:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574272.709482:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574272.709483:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574272.709486:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5421 00000100:00080000:3.0:1788574272.709487:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574272.709493:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574272.709501:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488751488:192.168.202.125@tcp:250:kworker.0 00000008:00100000:3.0:1788574272.709501:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574272.709519:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488751872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574272.709526:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574272.709529:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574272.709531:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574272.709533:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5421 00000100:00080000:3.0:1788574272.709536:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574272.709538:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574272.709541:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574272.709548:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488752000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574272.709596:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574272.709646:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574272/real 1788574272] req@ffff94d408c56680 x1875450488751488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574272.709663:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488751488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574272.709673:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574272.709677:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574272.709680:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488751488:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574272.709686:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488751616:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574272.709704:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574272.709713:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574272/real 1788574272] req@ffff94d408c57b80 x1875450488751616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574272.709721:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488751616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574272.709727:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574272.709729:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574272.709732:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488751616:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574272.709756:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488751744:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574272.709769:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574272.709784:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574272/real 1788574272] req@ffff94d408c54a80 x1875450488751744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574272.709793:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488751744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574272.709799:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574272.709801:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574272.709804:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488751744:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574272.709807:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488751872:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574272.709820:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574272.709827:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574272/real 1788574272] req@ffff94d408c56d80 x1875450488751872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574272.709836:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488751872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574272.709845:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574272.709848:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574272.709852:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488751872:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574272.709859:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488752000:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574272.709881:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574272.709894:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574272/real 1788574272] req@ffff94d408c54e00 x1875450488752000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574272.709906:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488752000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574272.709912:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574272.709915:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574272.709917:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488752000:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574273.221268:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574273.669329:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574274.245316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574274.693348:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574275.269315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574275.717368:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574276.293314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574276.741361:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574277.317281:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574277.765321:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574277.829308:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574277.829317:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574277.829320:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574277.829329:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5426 00000100:00080000:2.0:1788574277.829338:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574277.829345:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574277.829380:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60e00 x1875450488752128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574277.829409:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574277.829412:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574277.829414:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574277.829416:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5426 00000100:00080000:2.0:1788574277.829419:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574277.829421:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574277.829429:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574277.829438:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63100 x1875450488752256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574277.829444:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574277.829449:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574277.829450:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574277.829452:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5426 00000100:00080000:2.0:1788574277.829454:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574277.829456:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574277.829459:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574277.829465:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488752384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574277.829471:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574277.829473:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574277.829474:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574277.829477:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5426 00000100:00080000:2.0:1788574277.829480:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574277.829482:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574277.829486:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574277.829492:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488752128:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574277.829494:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488752512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574277.829504:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574277.829506:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574277.829507:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574277.829510:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5426 00000100:00080000:2.0:1788574277.829512:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574277.829514:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574277.829518:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574277.829526:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488752640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574277.829556:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574277.829587:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574277/real 1788574277] req@ffff94d408c60e00 x1875450488752128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574277.829600:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60e00 x1875450488752128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574277.829608:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574277.829610:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574277.829613:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488752128:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574277.829618:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488752256:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574277.829631:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574277.829639:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574277/real 1788574277] req@ffff94d408c63100 x1875450488752256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574277.829647:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63100 x1875450488752256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574277.829656:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574277.829658:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574277.829661:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488752256:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574277.829664:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488752384:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574277.829674:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574277.829679:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574277/real 1788574277] req@ffff94d408c61880 x1875450488752384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574277.829698:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488752384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574277.829704:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574277.829706:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574277.829708:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488752384:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574277.829727:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488752512:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574277.829740:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574277.829747:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574277/real 1788574277] req@ffff94d408c61c00 x1875450488752512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574277.829755:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488752512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574277.829760:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574277.829762:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574277.829764:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488752512:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574277.829768:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488752640:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574277.829778:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574277.829783:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574277/real 1788574277] req@ffff94d408c61500 x1875450488752640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574277.829790:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488752640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574277.829795:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574277.829797:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574277.829799:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488752640:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574278.341292:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574278.789384:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574279.365288:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574279.813388:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574280.389309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574280.837384:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574281.413369:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574281.861378:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574282.437321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574282.885396:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574282.949269:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574282.949277:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574282.949281:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574282.949291:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5431 00000100:00080000:2.0:1788574282.949300:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574282.949307:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574282.949346:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488752768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574282.949374:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574282.949377:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574282.949378:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574282.949382:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5431 00000100:00080000:2.0:1788574282.949386:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574282.949390:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574282.949398:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574282.949412:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488752768:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574282.949412:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488752896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574282.949426:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574282.949429:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574282.949431:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574282.949435:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5431 00000100:00080000:2.0:1788574282.949438:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574282.949442:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574282.949446:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574282.949455:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488753024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574282.949462:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574282.949464:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574282.949466:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574282.949469:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5431 00000100:00080000:2.0:1788574282.949472:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574282.949475:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574282.949478:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574282.949486:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488753152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574282.949492:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574282.949495:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574282.949497:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574282.949500:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5431 00000100:00080000:2.0:1788574282.949503:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574282.949506:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574282.949509:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574282.949517:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488753280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574282.949534:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488752896:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574282.949564:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488753024:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574282.949611:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488753152:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574282.949644:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488753280:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574283.461299:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574283.909356:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574284.485317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574284.933364:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574285.509341:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574285.957384:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788574286.021498:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:1.0:1788574286.021553:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788574286.021572:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788574286.021618:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788574286.021629:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788574286.021638:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788574286.021649:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:0.0:1788574286.021678:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574282/real 1788574286] req@ffff94d408c61f80 x1875450488752896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574286.021713:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488752896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574286.021735:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574286.021741:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574286.021747:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488752896:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574286.021762:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574282/real 1788574286] req@ffff94d408c63480 x1875450488753024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574286.021777:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488753024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574286.021786:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574286.021790:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574286.021794:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488753024:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574286.021801:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574282/real 1788574286] req@ffff94d408c60a80 x1875450488753152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574286.021814:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488753152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574286.021821:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574286.021825:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574286.021828:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488753152:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574286.021836:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574282/real 1788574286] req@ffff94d408c63b80 x1875450488752768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574286.021850:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488752768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574286.021859:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574286.021864:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574286.021867:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488752768:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574286.021901:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574282/real 1788574286] req@ffff94d408c62a00 x1875450488753280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574286.021916:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488753280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574286.021925:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574286.021930:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574286.021932:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488753280:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574286.533345:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574286.981396:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574287.557318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574288.005352:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574288.069310:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574288.069319:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574288.069323:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574288.069332:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5436 00000100:00080000:3.0:1788574288.069341:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574288.069347:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574288.069384:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488753408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574288.069404:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574288.069407:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574288.069409:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574288.069411:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5436 00000100:00080000:3.0:1788574288.069414:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574288.069416:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574288.069424:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574288.069437:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488753536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574288.069449:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574288.069453:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574288.069457:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574288.069459:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488753408:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574288.069464:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5436 00000100:00080000:3.0:1788574288.069471:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574288.069476:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574288.069482:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574288.069496:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488753664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574288.069506:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574288.069510:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574288.069512:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574288.069516:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5436 00000100:00080000:3.0:1788574288.069521:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574288.069526:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:0.0:1788574288.069532:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:3.0:1788574288.069533:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574288.069550:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488753792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574288.069561:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574288.069565:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574288.069567:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574288.069573:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5436 00000100:00100000:0.0:1788574288.069575:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574288/real 1788574288] req@ffff94d408c55c00 x1875450488753408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574288.069578:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574288.069585:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574288.069591:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574288.069607:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488753920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574288.069615:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488753408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574288.069633:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574288.069638:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574288.069643:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488753408:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574288.069669:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488753536:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574288.069700:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574288.069716:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574288/real 1788574288] req@ffff94d408c57100 x1875450488753536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574288.069730:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488753536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574288.069740:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574288.069744:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574288.069748:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488753536:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574288.069764:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488753664:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574288.069788:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574288.069801:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574288/real 1788574288] req@ffff94d408c55180 x1875450488753664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574288.069815:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488753664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574288.069825:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574288.069829:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574288.069832:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488753664:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574288.069839:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488753792:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574288.069862:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574288.069874:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574288/real 1788574288] req@ffff94d408c57800 x1875450488753792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574288.069894:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488753792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574288.069902:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574288.069905:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574288.069909:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488753792:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574288.069915:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488753920:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574288.069954:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574288.069967:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574288/real 1788574288] req@ffff94d408c54380 x1875450488753920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574288.069980:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488753920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574288.069988:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574288.069991:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574288.069995:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488753920:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574288.581265:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574289.029333:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574289.605343:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574290.053364:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574290.629293:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574291.077348:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574291.653638:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574292.101368:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574292.677386:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574293.125381:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574293.189352:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574293.189360:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574293.189364:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574293.189372:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5441 00000100:00080000:2.0:1788574293.189381:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574293.189387:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574293.189426:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488754048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574293.189452:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574293.189455:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574293.189456:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574293.189459:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5441 00000100:00080000:2.0:1788574293.189462:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574293.189464:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574293.189472:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574293.189480:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488754176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574293.189488:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574293.189491:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574293.189492:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574293.189495:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5441 00000100:00080000:2.0:1788574293.189497:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574293.189503:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574293.189505:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488754048:192.168.202.125@tcp:250:kworker.0 00000008:00100000:2.0:1788574293.189508:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574293.189517:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488754304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574293.189526:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574293.189528:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574293.189529:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574293.189532:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5441 00000100:00080000:2.0:1788574293.189534:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574293.189537:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574293.189540:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574293.189548:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488754432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574293.189555:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574293.189557:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574293.189559:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574293.189561:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5441 00000100:00080000:2.0:1788574293.189564:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788574293.189568:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788574293.189571:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574293.189576:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574293.189585:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488754560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574293.189607:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574293/real 1788574293] req@ffff94d408c62d80 x1875450488754048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574293.189624:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488754048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574293.189636:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574293.189639:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574293.189642:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488754048:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574293.189646:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488754176:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574293.189661:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574293.189668:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574293/real 1788574293] req@ffff94d408c60000 x1875450488754176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574293.189676:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488754176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574293.189681:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574293.189684:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574293.189686:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488754176:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574293.189703:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488754304:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574293.189714:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574293.189720:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574293/real 1788574293] req@ffff94d408c63800 x1875450488754304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574293.189727:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488754304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574293.189732:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574293.189734:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574293.189736:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488754304:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574293.189740:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488754432:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574293.189749:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574293.189755:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574293/real 1788574293] req@ffff94d408c62680 x1875450488754432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574293.189762:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488754432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574293.189767:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574293.189769:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574293.189770:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488754432:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574293.189774:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488754560:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574293.189782:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574293.189788:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574293/real 1788574293] req@ffff94d408c61180 x1875450488754560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574293.189794:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488754560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574293.189799:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574293.189801:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574293.189803:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488754560:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574293.701558:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574294.149403:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574294.725344:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574295.173374:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574295.749391:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574296.197387:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574296.773617:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574297.221319:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574297.797303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574298.245403:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574298.309311:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574298.309320:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574298.309324:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574298.309332:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5446 00000100:00080000:0.0:1788574298.309342:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574298.309357:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574298.309391:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ed80 x1875450488754688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574298.309415:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574298.309418:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574298.309419:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574298.309422:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5446 00000100:00080000:0.0:1788574298.309425:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574298.309428:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574298.309436:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574298.309447:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488754816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574298.309449:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488754688:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788574298.309463:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574298.309466:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574298.309467:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574298.309470:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5446 00000100:00080000:0.0:1788574298.309473:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574298.309475:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574298.309478:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574298.309488:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488754944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574298.309501:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574298.309505:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574298.309507:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788574298.309510:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788574298.309514:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5446 00000100:00080000:0.0:1788574298.309520:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574298.309524:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574298.309531:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574298.309543:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574298/real 1788574298] req@ffff94d408d3ed80 x1875450488754688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574298.309547:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488755072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574298.309564:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ed80 x1875450488754688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574298.309565:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574298.309574:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574298.309578:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574298.309583:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5446 00000100:00080000:2.0:1788574298.309583:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574298.309591:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574298.309591:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:0.0:1788574298.309599:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574298.309600:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488754688:192.168.202.125@tcp:250:kworker.0 00000008:00100000:0.0:1788574298.309608:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574298.309624:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488755200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574298.309624:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488754816:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574298.309648:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574298.309659:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574298/real 1788574298] req@ffff94d408d3d880 x1875450488754816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574298.309668:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488754816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574298.309674:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574298.309676:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574298.309679:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488754816:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574298.309682:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488754944:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574298.309692:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574298.309698:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574298/real 1788574298] req@ffff94d408d3dc00 x1875450488754944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574298.309706:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488754944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574298.309715:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574298.309717:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574298.309719:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488754944:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574298.309731:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488755072:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574298.309743:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574298.309750:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574298/real 1788574298] req@ffff94d408d3e300 x1875450488755072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574298.309757:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488755072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574298.309761:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574298.309763:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574298.309765:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488755072:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574298.309788:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488755200:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574298.309801:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574298.309808:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574298/real 1788574298] req@ffff94d408d3c700 x1875450488755200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574298.309815:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488755200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574298.309820:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574298.309822:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574298.309825:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488755200:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574298.821295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574299.269398:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574299.845330:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574300.293407:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574300.869291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574301.317334:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574301.893320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574302.341343:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574302.917324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574303.365455:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574303.429334:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574303.429345:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574303.429348:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574303.429357:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5451 00000100:00080000:3.0:1788574303.429367:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574303.429373:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574303.429405:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488755328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574303.429435:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574303.429439:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574303.429440:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574303.429442:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5451 00000100:00080000:3.0:1788574303.429444:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574303.429447:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574303.429453:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574303.429463:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488755456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574303.429470:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574303.429473:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574303.429474:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574303.429477:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5451 00000100:00080000:3.0:1788574303.429480:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574303.429482:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574303.429486:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574303.429491:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488755328:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574303.429495:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488755584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574303.429503:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574303.429505:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574303.429507:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574303.429509:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5451 00000100:00080000:3.0:1788574303.429512:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574303.429515:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574303.429518:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574303.429526:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488755712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574303.429532:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574303.429534:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574303.429535:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574303.429538:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5451 00000100:00080000:3.0:1788574303.429541:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574303.429543:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574303.429546:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574303.429549:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574303.429556:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488755840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574303.429583:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574303/real 1788574303] req@ffff94d408c55880 x1875450488755328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574303.429604:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488755328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574303.429616:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574303.429620:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574303.429625:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488755328:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574303.429632:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488755456:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574303.429654:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574303.429667:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574303/real 1788574303] req@ffff94d408c56a00 x1875450488755456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574303.429678:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488755456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574303.429684:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574303.429686:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574303.429688:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488755456:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574303.429706:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488755584:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574303.429720:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574303.429727:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574303/real 1788574303] req@ffff94d408c57480 x1875450488755584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574303.429735:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488755584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574303.429740:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574303.429742:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574303.429745:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488755584:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574303.429748:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488755712:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574303.429769:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574303.429776:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574303/real 1788574303] req@ffff94d408c54700 x1875450488755712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574303.429783:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488755712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574303.429788:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574303.429790:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574303.429792:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488755712:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574303.429796:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488755840:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574303.429805:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574303.429811:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574303/real 1788574303] req@ffff94d408c56300 x1875450488755840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574303.429840:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488755840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574303.429846:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574303.429848:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574303.429850:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488755840:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574303.941294:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574304.389384:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574304.965283:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574305.413378:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574305.989289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574306.437362:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574307.013288:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574307.461346:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574308.037314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574308.485331:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574308.549298:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574308.549306:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574308.549310:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574308.549318:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5456 00000100:00080000:3.0:1788574308.549327:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574308.549333:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574308.549369:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488755968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574308.549400:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574308.549404:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574308.549406:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574308.549411:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5456 00000100:00080000:3.0:1788574308.549415:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574308.549419:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574308.549428:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574308.549439:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488756096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574308.549449:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574308.549454:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788574308.549455:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488755968:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574308.549456:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574308.549464:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5456 00000100:00080000:3.0:1788574308.549467:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574308.549469:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574308.549474:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574308.549489:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488756224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574308.549499:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574308.549503:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574308.549505:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574308.549510:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5456 00000100:00080000:3.0:1788574308.549515:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574308.549522:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:2.0:1788574308.549523:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:3.0:1788574308.549529:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574308.549541:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488756352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574308.549549:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574308.549551:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574308.549553:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574308.549556:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5456 00000100:00080000:3.0:1788574308.549559:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788574308.549561:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574308/real 1788574308] req@ffff94d408c54000 x1875450488755968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574308.549565:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574308.549571:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574308.549584:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488756480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574308.549593:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488755968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574308.549602:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574308.549606:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574308.549610:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488755968:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574308.549629:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488756096:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574308.549646:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574308.549654:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574308/real 1788574308] req@ffff94d408c55f80 x1875450488756096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574308.549665:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488756096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574308.549674:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574308.549677:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574308.549681:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488756096:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574308.549687:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488756224:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574308.549709:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574308.549723:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574308/real 1788574308] req@ffff94d408c56300 x1875450488756224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574308.549736:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488756224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574308.549745:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574308.549748:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574308.549752:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488756224:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574308.549759:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488756352:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574308.549778:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574308.549788:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574308/real 1788574308] req@ffff94d408c54700 x1875450488756352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574308.549800:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488756352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574308.549818:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574308.549821:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574308.549824:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488756352:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574308.549831:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488756480:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574308.549849:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574308.549861:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574308/real 1788574308] req@ffff94d408c57480 x1875450488756480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574308.549906:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488756480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574308.549914:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574308.549918:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574308.549921:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488756480:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574309.061302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574309.509352:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574310.085326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574310.533405:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574311.109334:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574311.557342:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574312.133288:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574312.581327:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574313.157302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574313.605355:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574313.669349:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574313.669357:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574313.669361:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574313.669369:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5461 00000100:00080000:2.0:1788574313.669378:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574313.669384:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574313.669418:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62300 x1875450488756608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574313.669441:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574313.669444:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574313.669445:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574313.669448:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5461 00000100:00080000:2.0:1788574313.669450:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574313.669453:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574313.669460:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574313.669469:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60380 x1875450488756736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574313.669482:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788574313.669485:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488756608:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788574313.669486:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574313.669490:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574313.669497:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5461 00000100:00080000:2.0:1788574313.669500:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574313.669504:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574313.669509:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574313.669524:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60700 x1875450488756864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574313.669534:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574313.669538:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574313.669541:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574313.669545:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5461 00000800:00000100:0.0:1788574313.669549:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788574313.669553:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574313.669558:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574313.669564:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574313.669576:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488756992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574313.669583:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574313/real 1788574313] req@ffff94d408c62300 x1875450488756608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574313.669585:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574313.669593:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574313.669597:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574313.669604:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5461 00000100:00080000:2.0:1788574313.669610:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574313.669615:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574313.669618:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574313.669623:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62300 x1875450488756608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574313.669628:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488757120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574313.669636:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574313.669642:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574313.669646:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488756608:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574313.669666:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488756736:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574313.669681:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574313.669690:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574313/real 1788574313] req@ffff94d408c60380 x1875450488756736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574313.669698:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60380 x1875450488756736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574313.669703:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574313.669706:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574313.669708:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488756736:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574313.669711:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488756864:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574313.669720:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574313.669726:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574313/real 1788574313] req@ffff94d408c60700 x1875450488756864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574313.669733:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60700 x1875450488756864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574313.669742:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574313.669745:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574313.669747:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488756864:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574313.669750:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488756992:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574313.669766:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574313.669776:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574313/real 1788574313] req@ffff94d408c61180 x1875450488756992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574313.669789:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488756992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574313.669797:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574313.669801:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574313.669804:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488756992:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574313.669811:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488757120:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574313.669828:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574313.669836:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574313/real 1788574313] req@ffff94d408c62680 x1875450488757120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574313.669844:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488757120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574313.669849:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574313.669851:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574313.669854:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488757120:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574314.181282:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574314.629329:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574315.205322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574315.653443:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574316.229317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574316.677370:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574317.253304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574317.701376:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574318.277294:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574318.725352:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574318.789283:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574318.789291:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574318.789295:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574318.789303:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5467 00000100:00080000:2.0:1788574318.789318:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574318.789324:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574318.789357:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488757248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574318.789385:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574318.789388:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574318.789389:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574318.789392:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5467 00000100:00080000:2.0:1788574318.789395:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574318.789398:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574318.789405:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574318.789417:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488757376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574318.789425:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574318.789430:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574318.789432:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574318.789435:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5467 00000100:00080000:2.0:1788574318.789438:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574318.789440:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574318.789443:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574318.789454:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488757504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574318.789456:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488757248:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788574318.789465:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574318.789468:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574318.789469:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574318.789472:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5467 00000100:00080000:2.0:1788574318.789474:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574318.789476:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574318.789480:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574318.789489:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488757632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574318.789495:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574318.789498:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574318.789500:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574318.789503:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5467 00000100:00080000:2.0:1788574318.789521:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574318.789524:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574318.789529:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574318.789535:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574318.789540:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488757760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574318.789588:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574318/real 1788574318] req@ffff94d408c63800 x1875450488757248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574318.789703:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488757248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574318.789716:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574318.789855:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574318.789862:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488757248:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574318.789874:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488757376:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574318.789991:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574318.790010:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574318/real 1788574318] req@ffff94d408c60000 x1875450488757376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574318.790026:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488757376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574318.790036:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574318.790040:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574318.790044:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488757376:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574318.790073:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488757504:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574318.790098:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574318.790112:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574318/real 1788574318] req@ffff94d408c62d80 x1875450488757504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574318.790127:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488757504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574318.790136:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574318.790140:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574318.790144:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488757504:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574318.790150:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488757632:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574318.790173:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574318.790187:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574318/real 1788574318] req@ffff94d408c62a00 x1875450488757632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574318.790200:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488757632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574318.790244:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574318.790249:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574318.790252:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488757632:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574318.790259:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488757760:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574318.790283:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574318.790297:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574318/real 1788574318] req@ffff94d408c63b80 x1875450488757760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574318.790311:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488757760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574318.790320:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574318.790323:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574318.790327:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488757760:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574319.301355:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574319.749395:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574320.325290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574320.773365:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574321.349281:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574321.797389:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574322.373347:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574322.821435:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574323.397351:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574323.845424:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574323.909284:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574323.909292:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574323.909295:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574323.909303:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5472 00000100:00080000:0.0:1788574323.909312:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574323.909318:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574323.909358:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488757888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574323.909378:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574323.909380:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574323.909382:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574323.909384:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5472 00000100:00080000:0.0:1788574323.909386:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574323.909389:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574323.909395:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574323.909403:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488758016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574323.909410:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574323.909413:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574323.909414:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574323.909417:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5472 00000100:00080000:0.0:1788574323.909420:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574323.909423:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574323.909426:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574323.909433:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488758144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574323.909442:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574323.909444:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574323.909445:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574323.909448:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5472 00000100:00080000:0.0:1788574323.909449:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574323.909454:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574323.909464:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488757888:192.168.202.125@tcp:250:kworker.0 00000008:00100000:0.0:1788574323.909465:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574323.909483:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488758272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574323.909491:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574323.909493:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574323.909495:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574323.909499:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5472 00000100:00080000:0.0:1788574323.909502:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574323.909504:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574323.909507:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574323.909517:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488758400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574323.909556:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574323.909598:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574323/real 1788574323] req@ffff94d408d3f480 x1875450488757888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574323.909616:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488757888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574323.909626:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574323.909630:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574323.909634:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488757888:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574323.909640:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488758016:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574323.909658:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574323.909666:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574323/real 1788574323] req@ffff94d408d3c380 x1875450488758016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574323.909675:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488758016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574323.909681:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574323.909683:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574323.909686:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488758016:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574323.909710:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488758144:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574323.909720:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574323.909726:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574323/real 1788574323] req@ffff94d408d3c000 x1875450488758144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574323.909734:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488758144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574323.909740:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574323.909742:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574323.909743:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488758144:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574323.909747:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488758272:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574323.909758:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574323.909764:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574323/real 1788574323] req@ffff94d408d3d180 x1875450488758272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574323.909776:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488758272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574323.909781:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574323.909783:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574323.909785:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488758272:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574323.909789:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488758400:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574323.909798:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574323.909804:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574323/real 1788574323] req@ffff94d408d3df80 x1875450488758400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574323.909811:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488758400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574323.909816:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574323.909818:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574323.909820:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488758400:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574324.421304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574324.869374:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574325.445343:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574325.893362:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574326.469324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574326.917403:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574327.493329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574327.941354:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574328.517276:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574328.965348:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574329.029310:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574329.029320:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574329.029325:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574329.029335:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5477 00000100:00080000:3.0:1788574329.029346:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574329.029354:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574329.029389:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488758528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574329.029411:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574329.029414:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574329.029415:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574329.029418:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5477 00000100:00080000:3.0:1788574329.029421:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574329.029425:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574329.029432:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574329.029440:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488758656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574329.029444:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488758528:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574329.029452:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574329.029455:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574329.029456:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574329.029459:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5477 00000100:00080000:3.0:1788574329.029461:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574329.029463:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574329.029466:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574329.029474:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488758784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574329.029485:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574329.029487:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574329.029489:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574329.029491:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5477 00000100:00080000:3.0:1788574329.029494:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574329.029496:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574329.029499:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574329.029506:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488758912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574329.029512:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788574329.029515:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574329.029517:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574329.029519:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574329.029522:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5477 00000100:00080000:3.0:1788574329.029525:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574329.029528:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574329.029531:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574329.029538:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488759040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574329.029552:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574329/real 1788574329] req@ffff94d408c56a00 x1875450488758528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574329.029568:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488758528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574329.029576:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574329.029580:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574329.029585:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488758528:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574329.029607:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488758656:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574329.029630:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574329.029640:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574329/real 1788574329] req@ffff94d408c55880 x1875450488758656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574329.029649:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488758656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574329.029655:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574329.029657:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574329.029659:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488758656:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574329.029663:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488758784:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574329.029674:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574329.029680:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574329/real 1788574329] req@ffff94d408c54380 x1875450488758784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574329.029692:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488758784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574329.029698:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574329.029700:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574329.029702:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488758784:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574329.029705:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488758912:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574329.029716:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574329.029722:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574329/real 1788574329] req@ffff94d408c57800 x1875450488758912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574329.029732:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488758912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574329.029740:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574329.029744:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574329.029747:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488758912:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574329.029754:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488759040:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574329.029774:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574329.029786:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574329/real 1788574329] req@ffff94d408c55180 x1875450488759040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574329.029800:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488759040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574329.029825:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574329.029829:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574329.029832:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488759040:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574329.541320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574329.989350:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574330.565322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574331.013374:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574331.589283:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574332.037386:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574332.613290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574333.061347:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574333.637290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574334.085341:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574334.149326:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574334.149334:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574334.149338:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574334.149346:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5482 00000100:00080000:2.0:1788574334.149355:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574334.149361:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574334.149396:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488759168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574334.149423:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574334.149428:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574334.149429:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574334.149434:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5482 00000100:00080000:2.0:1788574334.149438:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574334.149442:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574334.149453:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574334.149468:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488759296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574334.149478:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488759168:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788574334.149484:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574334.149489:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574334.149491:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574334.149496:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5482 00000100:00080000:2.0:1788574334.149501:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574334.149506:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574334.149511:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574334.149527:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488759424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574334.149537:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574334.149541:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574334.149542:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574334.149546:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5482 00000800:00000100:0.0:1788574334.149552:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788574334.149553:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574334.149560:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574334.149566:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574334.149588:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488759552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574334.149598:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574334.149602:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1788574334.149605:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574334/real 1788574334] req@ffff94d408c60a80 x1875450488759168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574334.149605:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574334.149614:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5482 00000100:00080000:2.0:1788574334.149620:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574334.149628:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574334.149633:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574334.149639:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488759168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574334.149651:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488759680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574334.149654:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574334.149663:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574334.149672:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488759168:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574334.149702:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488759296:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574334.149730:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574334.149745:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574334/real 1788574334] req@ffff94d408c63480 x1875450488759296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574334.149758:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488759296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574334.149764:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574334.149766:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574334.149768:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488759296:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574334.149773:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488759424:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574334.149788:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574334.149795:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574334/real 1788574334] req@ffff94d408c61f80 x1875450488759424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574334.149802:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488759424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574334.149808:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574334.149810:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574334.149813:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488759424:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574334.149818:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488759552:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574334.149837:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574334.149849:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574334/real 1788574334] req@ffff94d408c61500 x1875450488759552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574334.149861:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488759552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574334.149868:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574334.149872:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574334.149875:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488759552:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574334.149880:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488759680:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574334.149899:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574334.149910:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574334/real 1788574334] req@ffff94d408c61c00 x1875450488759680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574334.149922:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488759680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574334.149930:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574334.149932:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574334.149935:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488759680:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574334.661284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574335.109374:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574335.685305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574336.133360:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574336.709320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574337.157391:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574337.733330:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574338.181338:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574338.757316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574339.205366:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574339.269272:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574339.269279:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574339.269283:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574339.269292:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5487 00000100:00080000:2.0:1788574339.269301:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574339.269307:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574339.269338:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488759808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574339.269360:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574339.269363:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574339.269364:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574339.269366:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5487 00000100:00080000:2.0:1788574339.269369:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574339.269371:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574339.269378:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574339.269387:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63100 x1875450488759936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574339.269394:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574339.269397:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574339.269398:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574339.269401:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5487 00000100:00080000:2.0:1788574339.269403:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788574339.269408:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488759808:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788574339.269409:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574339.269418:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574339.269438:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60e00 x1875450488760064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574339.269445:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574339.269448:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574339.269449:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574339.269452:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5487 00000100:00080000:2.0:1788574339.269455:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574339.269458:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574339.269461:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574339.269470:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488760192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574339.269476:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574339.269478:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574339.269479:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788574339.269485:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788574339.269485:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5487 00000100:00080000:2.0:1788574339.269492:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574339.269494:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574339.269497:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574339.269505:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488760320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574339.269540:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574339/real 1788574339] req@ffff94d408c61880 x1875450488759808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574339.269561:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488759808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574339.269572:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574339.269576:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574339.269581:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488759808:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574339.269588:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488759936:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574339.269613:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574339.269626:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574339/real 1788574339] req@ffff94d408c63100 x1875450488759936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574339.269640:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63100 x1875450488759936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574339.269667:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574339.269671:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574339.269675:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488759936:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574339.269700:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488760064:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574339.269722:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574339.269745:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574339/real 1788574339] req@ffff94d408c60e00 x1875450488760064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574339.269758:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60e00 x1875450488760064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574339.269767:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574339.269770:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574339.269774:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488760064:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574339.269780:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488760192:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574339.269800:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574339.269812:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574339/real 1788574339] req@ffff94d408c61c00 x1875450488760192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574339.269835:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488760192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574339.269844:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574339.269847:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574339.269851:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488760192:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574339.269858:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488760320:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574339.269879:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574339.269890:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574339/real 1788574339] req@ffff94d408c61500 x1875450488760320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574339.269902:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488760320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574339.269910:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574339.269913:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574339.269917:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488760320:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574339.781310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574340.229400:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574340.805342:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574341.253406:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574341.829314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574342.277361:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574342.853307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574343.301385:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574343.877355:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574344.325369:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574344.389356:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574344.389370:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574344.389375:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574344.389389:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5492 00000100:00080000:3.0:1788574344.389403:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574344.389413:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574344.389463:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488760448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574344.389495:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574344.389500:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574344.389502:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574344.389507:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5492 00000100:00080000:3.0:1788574344.389510:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574344.389516:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574344.389528:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574344.389545:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488760576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574344.389553:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488760448:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574344.389564:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574344.389569:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574344.389572:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574344.389577:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5492 00000100:00080000:3.0:1788574344.389582:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574344.389586:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574344.389592:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574344.389607:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488760704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574344.389619:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574344.389623:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574344.389625:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574344.389630:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5492 00000800:00000100:0.0:1788574344.389635:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574344.389637:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574344.389644:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574344.389650:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574344.389662:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488760832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574344.389674:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574344.389680:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1788574344.389681:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574344/real 1788574344] req@ffff94d408c57100 x1875450488760448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574344.389684:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574344.389693:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5492 00000100:00080000:3.0:1788574344.389699:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574344.389703:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574344.389707:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488760448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788574344.389711:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788574344.389719:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574344.389723:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574344.389728:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488760960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574344.389730:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488760448:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574344.389770:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488760576:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574344.389797:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574344.389809:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574344/real 1788574344] req@ffff94d408c55c00 x1875450488760576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574344.389818:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488760576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574344.389825:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574344.389827:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574344.389829:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488760576:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574344.389833:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488760704:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574344.389846:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574344.389856:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574344/real 1788574344] req@ffff94d408c54e00 x1875450488760704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574344.389869:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488760704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574344.389878:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574344.389882:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574344.389886:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488760704:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574344.389893:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488760832:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574344.389914:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574344.389922:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574344/real 1788574344] req@ffff94d408c56d80 x1875450488760832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574344.389930:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488760832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574344.389936:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574344.389939:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574344.389941:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488760832:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574344.389945:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488760960:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574344.389956:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574344.389962:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574344/real 1788574344] req@ffff94d408c54a80 x1875450488760960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574344.389969:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488760960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574344.389974:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574344.389976:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574344.389978:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488760960:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574344.901323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574345.349391:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574345.925299:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574346.373409:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574346.949259:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574347.397370:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574347.973325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574348.421393:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574348.997297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574349.445420:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574349.509330:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574349.509356:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574349.509360:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574349.509368:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5497 00000100:00080000:0.0:1788574349.509377:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574349.509383:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574349.509415:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488761088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574349.509445:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574349.509453:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574349.509455:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574349.509459:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5497 00000100:00080000:0.0:1788574349.509461:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574349.509465:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574349.509472:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574349.509479:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488761088:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574349.509483:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488761216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574349.509493:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574349.509495:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574349.509497:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574349.509500:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5497 00000100:00080000:0.0:1788574349.509503:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574349.509505:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574349.509508:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574349.509516:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488761344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574349.509522:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574349.509525:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574349.509526:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574349.509529:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5497 00000100:00080000:0.0:1788574349.509531:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574349.509533:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574349.509536:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574349.509543:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488761472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574349.509549:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574349.509552:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574349.509553:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574349.509555:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5497 00000100:00080000:0.0:1788574349.509557:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574349.509560:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574349.509563:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574349.509568:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488761600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574349.509589:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488761216:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574349.509618:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488761344:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574349.509631:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488761472:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574349.509644:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488761600:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574350.021250:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574350.469356:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574351.045310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574351.493385:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574352.069261:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574352.517369:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788574352.581447:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:2.0:1788574352.581492:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788574352.581509:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788574352.581545:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788574352.581556:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788574352.581567:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788574352.581579:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:3.0:1788574352.581594:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574349/real 1788574352] req@ffff94d408d3ea00 x1875450488761216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574352.581622:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488761216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574352.581635:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574352.581640:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574352.581644:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488761216:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574352.581653:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574349/real 1788574352] req@ffff94d408d3e680 x1875450488761344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574352.581668:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488761344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574352.581677:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574352.581681:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574352.581684:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488761344:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574352.581692:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574349/real 1788574352] req@ffff94d408d3d500 x1875450488761472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574352.581705:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488761472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574352.581714:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574352.581718:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574352.581721:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488761472:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574352.581731:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574349/real 1788574352] req@ffff94d408d3ca80 x1875450488761600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574352.581742:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488761600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574352.581750:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574352.581753:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574352.581757:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488761600:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574352.581764:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574349/real 1788574352] req@ffff94d408d3ce00 x1875450488761088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574352.581777:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488761088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574352.581785:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574352.581789:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574352.581792:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488761088:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574353.093314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574353.541391:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574354.117275:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574354.565342:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574354.629315:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574354.629326:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574354.629331:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574354.629350:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5502 00000100:00080000:3.0:1788574354.629361:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574354.629370:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574354.629405:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488761728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574354.629439:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574354.629443:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574354.629446:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574354.629451:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5502 00000100:00080000:3.0:1788574354.629455:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574354.629461:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574354.629470:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574354.629487:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488761856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574354.629492:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488761728:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574354.629506:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574354.629511:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574354.629513:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574354.629518:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5502 00000100:00080000:3.0:1788574354.629522:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574354.629527:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574354.629532:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574354.629547:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488761984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574354.629558:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574354.629561:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574354.629564:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788574354.629568:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574354.629572:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5502 00000100:00080000:3.0:1788574354.629578:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574354.629583:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574354.629589:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574354.629604:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488762112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574354.629613:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574354/real 1788574354] req@ffff94d408c57b80 x1875450488761728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574354.629616:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574354.629623:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574354.629627:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574354.629637:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5502 00000100:00080000:3.0:1788574354.629642:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574354.629647:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574354.629650:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488761728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788574354.629653:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788574354.629668:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788574354.629672:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488762240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574354.629676:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574354.629684:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488761728:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574354.629715:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488761856:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574354.629742:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574354.629757:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574354/real 1788574354] req@ffff94d408c56680 x1875450488761856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574354.629772:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488761856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574354.629780:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574354.629784:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574354.629788:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488761856:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574354.629802:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488761984:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574354.629825:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574354.629838:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574354/real 1788574354] req@ffff94d408c54a80 x1875450488761984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574354.629852:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488761984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574354.629861:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574354.629864:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574354.629868:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488761984:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574354.629875:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488762112:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574354.629898:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574354.629911:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574354/real 1788574354] req@ffff94d408c56d80 x1875450488762112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574354.629932:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488762112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574354.629942:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574354.629945:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574354.629949:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488762112:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574354.629955:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488762240:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574354.629977:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574354.629989:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574354/real 1788574354] req@ffff94d408c54e00 x1875450488762240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574354.630002:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488762240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574354.630011:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574354.630014:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574354.630018:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488762240:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574355.141349:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574355.589374:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574356.165332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574356.613391:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574357.189368:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574357.637370:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574358.213421:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574358.661434:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574359.237838:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574359.685345:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574359.749311:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574359.749319:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574359.749323:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574359.749331:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5508 00000100:00080000:2.0:1788574359.749340:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574359.749346:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574359.749395:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488762368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574359.749419:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574359.749421:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574359.749423:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574359.749425:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5508 00000100:00080000:2.0:1788574359.749428:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574359.749430:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574359.749442:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574359.749453:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488762496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574359.749461:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574359.749463:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574359.749467:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574359.749469:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488762368:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788574359.749473:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5508 00000100:00080000:2.0:1788574359.749478:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574359.749480:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574359.749484:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574359.749498:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488762624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574359.749505:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574359.749507:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574359.749508:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574359.749510:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5508 00000100:00080000:2.0:1788574359.749513:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574359.749515:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574359.749519:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574359.749526:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488762752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574359.749533:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574359.749535:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574359.749537:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788574359.749541:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788574359.749545:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5508 00000100:00080000:2.0:1788574359.749549:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574359.749551:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574359.749555:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574359.749564:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488762880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574359.749586:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574359/real 1788574359] req@ffff94d408c61f80 x1875450488762368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574359.749608:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488762368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574359.749623:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574359.749626:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574359.749629:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488762368:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574359.749648:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488762496:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574359.749667:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574359.749677:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574359/real 1788574359] req@ffff94d408c63480 x1875450488762496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574359.749685:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488762496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574359.749691:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574359.749693:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574359.749695:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488762496:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574359.749698:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488762624:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574359.749709:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574359.749714:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574359/real 1788574359] req@ffff94d408c60a80 x1875450488762624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574359.749722:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488762624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574359.749727:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574359.749729:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574359.749731:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488762624:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574359.749735:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488762752:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574359.749745:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574359.749750:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574359/real 1788574359] req@ffff94d408c63b80 x1875450488762752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574359.749758:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488762752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574359.749763:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574359.749764:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574359.749766:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488762752:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574359.749770:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488762880:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574359.749779:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574359.749784:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574359/real 1788574359] req@ffff94d408c62a00 x1875450488762880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574359.749791:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488762880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574359.749796:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574359.749797:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574359.749799:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488762880:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574360.261645:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574360.709412:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574361.285549:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574361.733388:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574362.309331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574362.757409:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574363.333972:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574363.781441:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574364.357290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574364.805380:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574364.869335:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574364.869345:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574364.869350:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574364.869360:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5513 00000100:00080000:3.0:1788574364.869371:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574364.869380:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574364.869432:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488763008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574364.869460:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574364.869465:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574364.869467:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574364.869471:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5513 00000100:00080000:3.0:1788574364.869476:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574364.869482:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574364.869499:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574364.869512:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488763008:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574364.869516:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488763136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574364.869531:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574364.869535:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574364.869538:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574364.869542:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5513 00000100:00080000:3.0:1788574364.869547:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574364.869552:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574364.869558:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574364.869571:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488763264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574364.869578:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574364.869588:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574364.869593:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574364.869595:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574364.869601:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5513 00000100:00080000:3.0:1788574364.869606:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574364.869610:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574364.869617:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574364.869623:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574364/real 1788574364] req@ffff94d408c55c00 x1875450488763008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574364.869631:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488763392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574364.869646:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574364.869649:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574364.869650:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574364.869654:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488763008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574364.869656:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5513 00000100:00080000:3.0:1788574364.869662:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574364.869669:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00080000:0.0:1788574364.869670:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788574364.869674:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788574364.869676:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574364.869682:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488763008:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574364.869687:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488763520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574364.869706:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488763136:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574364.869735:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574364.869748:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574364/real 1788574364] req@ffff94d408c57100 x1875450488763136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574364.869757:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488763136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574364.869762:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574364.869764:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574364.869767:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488763136:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574364.869770:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488763264:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574364.869782:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574364.869788:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574364/real 1788574364] req@ffff94d408c55180 x1875450488763264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574364.869796:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488763264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574364.869805:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574364.869808:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574364.869810:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488763264:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574364.869813:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488763392:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574364.869823:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574364.869828:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574364/real 1788574364] req@ffff94d408c57800 x1875450488763392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574364.869835:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488763392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574364.869840:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574364.869842:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574364.869844:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488763392:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574364.869847:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488763520:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574364.869861:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574364.869871:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574364/real 1788574364] req@ffff94d408c54380 x1875450488763520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574364.869884:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488763520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574364.869893:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574364.869896:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574364.869900:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488763520:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574365.381289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574365.829363:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574366.405330:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574366.853442:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574367.429279:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574367.877359:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574368.453305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574368.901384:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574369.477291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574369.925342:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574369.989276:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574369.989285:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574369.989289:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574369.989300:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5518 00000100:00080000:3.0:1788574369.989310:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574369.989317:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574369.989351:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488763648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574369.989377:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574369.989381:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574369.989383:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574369.989387:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5518 00000100:00080000:3.0:1788574369.989391:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574369.989395:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574369.989403:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574369.989417:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488763776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574369.989421:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488763648:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574369.989438:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574369.989442:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574369.989445:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574369.989451:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5518 00000100:00080000:3.0:1788574369.989482:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574369.989487:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574369.989493:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788574369.989499:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574369.989508:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488763904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574369.989520:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574369.989524:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574369.989526:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574369.989531:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5518 00000100:00100000:0.0:1788574369.989535:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574369/real 1788574369] req@ffff94d408c55880 x1875450488763648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574369.989537:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574369.989545:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574369.989552:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574369.989564:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488763648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574369.989564:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488764032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574369.989585:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574369.989588:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574369.989590:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574369.989593:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574369.989594:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:3.0:1788574369.989604:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5518 00000100:00100000:0.0:1788574369.989605:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488763648:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574369.989611:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574369.989617:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574369.989622:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574369.989633:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488764160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574369.989637:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488763776:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574369.989674:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574369.989686:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574369/real 1788574369] req@ffff94d408c56a00 x1875450488763776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574369.989695:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488763776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574369.989702:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574369.989704:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574369.989706:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488763776:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574369.989710:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488763904:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574369.989723:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574369.989729:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574369/real 1788574369] req@ffff94d408c57480 x1875450488763904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574369.989737:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488763904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574369.989742:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574369.989744:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574369.989746:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488763904:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574369.989750:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488764032:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574369.989760:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574369.989766:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574369/real 1788574369] req@ffff94d408c54700 x1875450488764032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574369.989773:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488764032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574369.989778:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574369.989780:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574369.989782:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488764032:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574369.989792:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488764160:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574369.989801:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574369.989807:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574369/real 1788574369] req@ffff94d408c56300 x1875450488764160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574369.989814:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488764160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574369.989820:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574369.989822:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574369.989824:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488764160:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574370.501322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574370.949355:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574371.525295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574371.973348:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574372.549293:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574372.997360:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574373.573284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574374.021390:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574374.597277:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574375.045377:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574375.109325:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574375.109334:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574375.109339:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574375.109349:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5523 00000100:00080000:3.0:1788574375.109360:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574375.109368:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574375.109406:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488764288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574375.109437:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574375.109441:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574375.109443:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574375.109447:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5523 00000100:00080000:3.0:1788574375.109458:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574375.109463:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574375.109474:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574375.109484:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488764288:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574375.109490:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488764416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574375.109503:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574375.109521:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574375.109523:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574375.109528:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5523 00000100:00080000:3.0:1788574375.109532:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574375.109537:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574375.109542:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788574375.109556:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574375.109558:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488764544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574375.109570:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574375.109574:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574375.109576:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574375.109580:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5523 00000100:00080000:3.0:1788574375.109590:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788574375.109596:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574375/real 1788574375] req@ffff94d408c55f80 x1875450488764288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574375.109597:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574375.109606:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574375.109621:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488764672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574375.109629:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488764288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574375.109635:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574375.109648:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574375.109650:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574375.109652:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574375.109657:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5523 00000100:00080000:0.0:1788574375.109660:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:3.0:1788574375.109663:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788574375.109668:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488764288:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574375.109672:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574375.109680:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574375.109691:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488764800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574375.109703:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488764416:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574375.109728:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574375.109739:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574375/real 1788574375] req@ffff94d408c54000 x1875450488764416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574375.109748:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488764416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574375.109754:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574375.109756:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574375.109758:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488764416:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574375.109763:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488764544:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574375.109775:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574375.109781:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574375/real 1788574375] req@ffff94d408c56300 x1875450488764544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574375.109789:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488764544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574375.109795:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574375.109796:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574375.109798:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488764544:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574375.109802:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488764672:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574375.109813:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574375.109818:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574375/real 1788574375] req@ffff94d408c54700 x1875450488764672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574375.109825:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488764672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574375.109830:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574375.109832:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574375.109834:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488764672:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574375.109844:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488764800:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574375.109854:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574375.109860:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574375/real 1788574375] req@ffff94d408c57480 x1875450488764800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574375.109868:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488764800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574375.109872:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574375.109874:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574375.109876:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488764800:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574375.621300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574376.069373:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574376.645294:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574377.093360:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574377.669337:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574378.117374:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574378.693284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574379.141366:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574379.717302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574380.165358:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574380.229346:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574380.229354:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574380.229358:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574380.229367:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5528 00000100:00080000:2.0:1788574380.229377:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574380.229383:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574380.229415:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488764928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574380.229436:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574380.229439:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574380.229441:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574380.229443:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5528 00000100:00080000:2.0:1788574380.229446:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574380.229449:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574380.229456:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574380.229465:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488765056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574380.229474:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488764928:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788574380.229476:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574380.229482:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574380.229483:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574380.229486:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5528 00000100:00080000:2.0:1788574380.229488:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574380.229491:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574380.229495:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574380.229502:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488765184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574380.229509:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574380.229512:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574380.229513:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574380.229516:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5528 00000100:00080000:2.0:1788574380.229519:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574380.229522:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574380.229525:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574380.229532:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488765312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574380.229539:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788574380.229540:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788574380.229545:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574380.229547:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574380.229550:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5528 00000100:00080000:2.0:1788574380.229554:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574380.229557:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574380.229583:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574380.229593:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488765440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574380.229601:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574380/real 1788574380] req@ffff94d408c62d80 x1875450488764928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574380.229631:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488764928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574380.229642:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574380.229646:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574380.229657:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488764928:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574380.229682:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488765056:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574380.229706:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574380.229720:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574380/real 1788574380] req@ffff94d408c60000 x1875450488765056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574380.229732:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488765056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574380.229747:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574380.229750:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574380.229753:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488765056:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574380.229759:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488765184:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574380.229778:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574380.229789:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574380/real 1788574380] req@ffff94d408c63800 x1875450488765184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574380.229801:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488765184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574380.229809:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574380.229813:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574380.229816:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488765184:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574380.229822:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488765312:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574380.229840:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574380.229852:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574380/real 1788574380] req@ffff94d408c62680 x1875450488765312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574380.229863:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488765312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574380.229871:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574380.229874:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574380.229878:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488765312:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574380.229883:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488765440:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574380.229901:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574380.229911:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574380/real 1788574380] req@ffff94d408c61180 x1875450488765440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574380.229922:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488765440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574380.229931:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574380.229934:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574380.229937:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488765440:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574380.741275:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574381.189375:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574381.765312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574382.213390:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574382.789320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574383.237364:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574383.813325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574384.261394:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574384.837323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574385.285371:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574385.349305:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574385.349314:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574385.349318:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574385.349326:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5533 00000100:00080000:3.0:1788574385.349337:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574385.349345:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574385.349384:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488765568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574385.349408:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574385.349411:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574385.349417:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574385.349420:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5533 00000100:00080000:3.0:1788574385.349423:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574385.349427:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574385.349435:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574385.349446:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488765696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574385.349453:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488765568:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574385.349456:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574385.349460:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574385.349462:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574385.349465:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5533 00000100:00080000:3.0:1788574385.349467:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574385.349469:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574385.349472:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574385.349480:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488765824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574385.349487:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574385.349489:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574385.349490:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574385.349493:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5533 00000100:00080000:3.0:1788574385.349496:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574385.349498:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574385.349501:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574385.349509:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488765952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574385.349514:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574385.349523:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574385.349525:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574385.349527:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574385.349530:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5533 00000100:00080000:3.0:1788574385.349533:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574385.349536:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574385.349539:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574385.349545:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488766080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574385.349553:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574385/real 1788574385] req@ffff94d408c56a00 x1875450488765568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574385.349569:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488765568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574385.349578:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574385.349591:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574385.349596:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488765568:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574385.349628:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488765696:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574385.349647:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574385.349657:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574385/real 1788574385] req@ffff94d408c55880 x1875450488765696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574385.349665:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488765696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574385.349671:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574385.349673:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574385.349676:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488765696:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574385.349680:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488765824:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574385.349690:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574385.349697:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574385/real 1788574385] req@ffff94d408c54380 x1875450488765824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574385.349704:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488765824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574385.349709:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574385.349711:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574385.349713:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488765824:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574385.349716:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488765952:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574385.349726:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574385.349732:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574385/real 1788574385] req@ffff94d408c57800 x1875450488765952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574385.349739:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488765952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574385.349745:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574385.349747:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574385.349749:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488765952:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574385.349752:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488766080:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574385.349762:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574385.349767:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574385/real 1788574385] req@ffff94d408c55180 x1875450488766080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574385.349774:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488766080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574385.349779:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574385.349781:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574385.349782:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488766080:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574385.861290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574386.309356:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574386.885293:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574387.333320:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574387.909286:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574388.357366:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574388.933285:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574389.381358:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574389.957326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574390.405381:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574390.469299:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574390.469307:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574390.469316:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574390.469325:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5538 00000100:00080000:3.0:1788574390.469333:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574390.469340:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574390.469369:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488766208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574390.469396:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574390.469399:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574390.469400:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574390.469403:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5538 00000100:00080000:3.0:1788574390.469405:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574390.469408:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574390.469415:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574390.469426:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488766336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574390.469433:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574390.469436:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574390.469437:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574390.469441:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5538 00000100:00080000:3.0:1788574390.469443:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574390.469446:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574390.469449:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574390.469457:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488766464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574390.469466:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574390.469468:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574390.469470:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574390.469480:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5538 00000100:00100000:0.0:1788574390.469483:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488766208:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574390.469485:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574390.469497:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574390.469501:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574390.469523:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488766592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574390.469529:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574390.469532:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574390.469535:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788574390.469586:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574390.469593:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5538 00000100:00080000:3.0:1788574390.469600:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574390.469604:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574390.469609:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574390.469626:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488766720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574390.469644:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574390/real 1788574390] req@ffff94d408c57100 x1875450488766208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574390.469680:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488766208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574390.469690:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574390.469696:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574390.469699:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488766208:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574390.469708:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488766336:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574390.469741:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574390.469761:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574390/real 1788574390] req@ffff94d408c55c00 x1875450488766336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574390.469776:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488766336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574390.469782:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574390.469785:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574390.469789:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488766336:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574390.469826:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488766464:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574390.469871:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574390.469882:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574390/real 1788574390] req@ffff94d408c54e00 x1875450488766464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574390.469891:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488766464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574390.469897:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574390.469900:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574390.469902:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488766464:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574390.469906:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488766592:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574390.469920:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574390.469927:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574390/real 1788574390] req@ffff94d408c56d80 x1875450488766592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574390.469935:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488766592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574390.469941:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574390.469942:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574390.469944:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488766592:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574390.469948:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488766720:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574390.469959:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574390.469965:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574390/real 1788574390] req@ffff94d408c54a80 x1875450488766720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574390.469973:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488766720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574390.469978:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574390.469980:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574390.469982:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488766720:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574390.981295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574391.429396:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574392.005306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574392.453392:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574393.029293:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574393.477342:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574394.053304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574394.501371:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574395.077310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574395.525451:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574395.589355:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574395.589362:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574395.589366:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574395.589376:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5543 00000100:00080000:0.0:1788574395.589388:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574395.589396:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574395.589433:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f800 x1875450488766848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574395.589464:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574395.589467:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574395.589468:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574395.589471:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5543 00000100:00080000:0.0:1788574395.589474:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574395.589477:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574395.589484:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574395.589495:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f100 x1875450488766976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574395.589502:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574395.589505:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574395.589506:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574395.589509:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5543 00000100:00080000:0.0:1788574395.589511:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574395.589513:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574395.589516:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574395.589525:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488766848:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574395.589527:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488767104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574395.589552:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574395.589554:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574395.589555:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574395.589559:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5543 00000100:00080000:0.0:1788574395.589562:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574395.589565:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574395.589570:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574395.589577:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488767232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574395.589583:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574395.589586:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574395.589587:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574395.589590:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5543 00000100:00080000:0.0:1788574395.589597:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574395.589600:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574395.589603:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574395.589609:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488767360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574395.589627:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574395.589681:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574395/real 1788574395] req@ffff94d408d3f800 x1875450488766848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574395.589705:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f800 x1875450488766848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574395.589715:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574395.589734:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574395.589737:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488766848:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574395.589745:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488766976:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574395.589769:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574395.589784:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574395/real 1788574395] req@ffff94d408d3f100 x1875450488766976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574395.589814:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f100 x1875450488766976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574395.589822:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574395.589825:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574395.589827:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488766976:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574395.589853:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488767104:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574395.589870:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574395.589879:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574395/real 1788574395] req@ffff94d408d3ce00 x1875450488767104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574395.589887:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488767104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574395.589898:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574395.589900:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574395.589903:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488767104:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574395.589906:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488767232:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574395.589918:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574395.589924:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574395/real 1788574395] req@ffff94d408d3ca80 x1875450488767232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574395.589931:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488767232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574395.589937:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574395.589938:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574395.589940:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488767232:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574395.589944:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488767360:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574395.589954:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574395.589959:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574395/real 1788574395] req@ffff94d408d3d500 x1875450488767360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574395.589967:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488767360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574395.589971:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574395.589973:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574395.589976:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488767360:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574396.101289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574396.549370:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574397.125354:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574397.573397:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574398.149270:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574398.597368:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574399.173297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574399.621357:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574400.197291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574400.645358:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574400.709321:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574400.709331:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574400.709334:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574400.709343:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5549 00000100:00080000:3.0:1788574400.709352:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574400.709357:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574400.709390:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488767488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574400.709418:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574400.709421:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574400.709423:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574400.709426:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5549 00000100:00080000:3.0:1788574400.709429:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574400.709432:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574400.709440:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574400.709447:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488767488:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574400.709451:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488767616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574400.709466:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574400.709468:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574400.709470:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574400.709472:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5549 00000100:00080000:3.0:1788574400.709476:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574400.709478:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574400.709481:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574400.709490:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488767744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574400.709496:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574400.709499:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574400.709500:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788574400.709504:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574400.709507:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5549 00000100:00080000:3.0:1788574400.709512:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574400.709515:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574400.709519:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574400.709527:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488767872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574400.709534:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574400.709537:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574400.709538:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574400.709541:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574400/real 1788574400] req@ffff94d408c56680 x1875450488767488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574400.709544:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5549 00000100:00080000:3.0:1788574400.709548:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574400.709550:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574400.709553:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574400.709563:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488768000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574400.709571:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488767488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574400.709584:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574400.709589:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574400.709595:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488767488:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574400.709627:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488767616:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574400.709652:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574400.709666:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574400/real 1788574400] req@ffff94d408c57b80 x1875450488767616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574400.709679:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488767616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574400.709697:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574400.709701:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574400.709705:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488767616:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574400.709712:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488767744:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574400.709732:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574400.709743:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574400/real 1788574400] req@ffff94d408c54a80 x1875450488767744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574400.709756:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488767744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574400.709766:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574400.709783:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574400.709787:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488767744:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574400.709795:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488767872:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574400.709815:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574400.709827:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574400/real 1788574400] req@ffff94d408c56d80 x1875450488767872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574400.709840:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488767872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574400.709857:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574400.709861:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574400.709864:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488767872:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574400.709872:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488768000:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574400.709892:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574400.709903:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574400/real 1788574400] req@ffff94d408c54e00 x1875450488768000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574400.709916:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488768000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574400.709925:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574400.709928:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574400.709932:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488768000:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574401.221351:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574401.669388:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574402.245324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574402.693427:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574403.269363:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574403.717364:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574404.293317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574404.741337:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574405.317325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574405.765357:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574405.829282:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574405.829293:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574405.829298:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574405.829309:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5554 00000100:00080000:3.0:1788574405.829320:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574405.829328:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574405.829367:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488768128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574405.829398:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574405.829402:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574405.829404:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574405.829409:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5554 00000100:00080000:3.0:1788574405.829413:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574405.829419:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574405.829429:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574405.829440:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488768128:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574405.829449:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488768256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574405.829463:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574405.829466:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574405.829468:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574405.829471:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5554 00000100:00080000:3.0:1788574405.829474:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574405.829478:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574405.829482:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574405.829494:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488768384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574405.829499:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574405.829507:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574405.829510:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574405.829511:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574405.829515:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5554 00000100:00080000:3.0:1788574405.829526:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574405.829529:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574405.829532:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574405.829538:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574405/real 1788574405] req@ffff94d408c55c00 x1875450488768128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574405.829553:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488768512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574405.829554:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488768128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574405.829591:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574405.829592:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574405.829600:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:3.0:1788574405.829600:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574405.829607:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574405.829608:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488768128:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574405.829620:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5554 00000100:00080000:3.0:1788574405.829626:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574405.829632:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574405.829634:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488768256:192.168.202.125@tcp:38:kworker.0 00000008:00100000:3.0:1788574405.829641:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788574405.829658:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574405.829660:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488768640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574405.829670:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574405/real 1788574405] req@ffff94d408c57100 x1875450488768256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574405.829680:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488768256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574405.829686:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574405.829688:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574405.829691:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488768256:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574405.829695:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488768384:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574405.829705:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574405.829713:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574405/real 1788574405] req@ffff94d408c55180 x1875450488768384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574405.829725:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488768384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574405.829734:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574405.829737:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574405.829741:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488768384:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574405.829748:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488768512:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574405.829770:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574405.829782:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574405/real 1788574405] req@ffff94d408c57800 x1875450488768512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574405.829796:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488768512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574405.829805:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574405.829808:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574405.829812:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488768512:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574405.829845:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488768640:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574405.829867:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574405.829880:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574405/real 1788574405] req@ffff94d408c54380 x1875450488768640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574405.829893:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488768640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574405.829903:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574405.829907:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574405.829911:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488768640:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574406.341286:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574406.789321:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574407.365298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574407.813401:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574408.389295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574408.837378:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574409.413313:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574409.861376:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574410.437301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574410.885341:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574410.949271:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574410.949279:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574410.949282:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574410.949290:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5559 00000100:00080000:3.0:1788574410.949298:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574410.949305:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574410.949337:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488768768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574410.949362:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574410.949367:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574410.949368:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574410.949373:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5559 00000100:00080000:3.0:1788574410.949377:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574410.949383:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574410.949394:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574410.949398:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488768768:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574410.949410:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488768896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574410.949419:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574410.949422:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574410.949424:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574410.949427:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5559 00000100:00080000:3.0:1788574410.949430:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574410.949433:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574410.949444:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574410.949457:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488769024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574410.949462:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574410.949473:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574410.949478:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574410.949481:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574410.949486:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5559 00000100:00080000:3.0:1788574410.949491:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574410.949496:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574410.949502:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574410.949506:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574410/real 1788574410] req@ffff94d408c55880 x1875450488768768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574410.949514:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488769152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574410.949539:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488768768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574410.949543:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574410.949550:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574410.949553:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574410.949559:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5559 00000100:00080000:0.0:1788574410.949560:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574410.949563:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574410.949567:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:3.0:1788574410.949571:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574410.949574:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488768768:192.168.202.125@tcp:250:kworker.0 00000008:00100000:3.0:1788574410.949576:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574410.949586:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488769280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574410.949599:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488768896:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574410.949621:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574410.949630:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574410/real 1788574410] req@ffff94d408c56a00 x1875450488768896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574410.949639:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488768896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574410.949645:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574410.949647:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574410.949649:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488768896:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574410.949653:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488769024:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574410.949663:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574410.949669:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574410/real 1788574410] req@ffff94d408c57480 x1875450488769024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574410.949676:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488769024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574410.949686:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574410.949688:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574410.949690:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488769024:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574410.949694:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488769152:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574410.949704:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574410.949711:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574410/real 1788574410] req@ffff94d408c54700 x1875450488769152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574410.949722:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488769152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574410.949730:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574410.949734:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574410.949737:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488769152:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574410.949744:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488769280:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574410.949763:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574410.949773:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574410/real 1788574410] req@ffff94d408c56300 x1875450488769280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574410.949781:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488769280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574410.949787:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574410.949789:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574410.949791:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488769280:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574411.461317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574411.909379:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574412.485290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574412.933329:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574413.509322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574413.957385:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574414.533300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574414.981428:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574415.557326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574416.005415:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574416.069313:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574416.069327:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574416.069333:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574416.069346:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5564 00000100:00080000:3.0:1788574416.069361:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574416.069371:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574416.069438:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488769408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574416.069476:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574416.069482:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574416.069484:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574416.069489:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5564 00000100:00080000:3.0:1788574416.069493:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574416.069499:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574416.069512:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574416.069530:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488769536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574416.069544:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488769408:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574416.069551:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574416.069557:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574416.069560:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574416.069566:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5564 00000100:00080000:3.0:1788574416.069572:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574416.069577:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574416.069583:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574416.069602:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488769664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574416.069613:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574416.069619:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574416.069621:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574416.069625:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5564 00000100:00080000:3.0:1788574416.069629:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574416.069636:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574416.069644:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574416.069661:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488769792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574416.069677:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488769536:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788574416.069678:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574416.069686:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574416.069693:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574416.069702:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5564 00000100:00080000:3.0:1788574416.069708:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574416.069714:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574416.069720:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574416.069742:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488769664:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574416.069742:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488769920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574416.069805:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488769792:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574416.069843:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488769920:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574416.581285:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574417.029428:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574417.605312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574418.053381:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574418.629295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574419.077384:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788574419.141689:0:2392:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:0.0:1788574419.141741:0:2392:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788574419.141759:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:0.0:1788574419.141804:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:0.0:1788574419.141820:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:0.0:1788574419.141831:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:0.0:1788574419.141840:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:1.0:1788574419.141876:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574416/real 1788574419] req@ffff94d408c54700 x1875450488769792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574419.141906:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488769792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574419.141918:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574419.141924:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574419.141928:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488769792:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574419.141936:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574416/real 1788574419] req@ffff94d408c57480 x1875450488769920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574419.141949:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488769920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574419.141956:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574419.141973:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574419.141977:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488769920:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574419.141985:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574416/real 1788574419] req@ffff94d408c55f80 x1875450488769536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574419.141997:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488769536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574419.142003:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574419.142007:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574419.142010:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488769536:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574419.142016:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574416/real 1788574419] req@ffff94d408c56300 x1875450488769664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574419.142026:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488769664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574419.142033:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574419.142036:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574419.142039:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488769664:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574419.142047:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574416/real 1788574419] req@ffff94d408c54000 x1875450488769408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574419.142059:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488769408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574419.142067:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574419.142071:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574419.142074:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488769408:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574419.653289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574420.101434:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574420.677367:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574421.125390:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574421.189321:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574421.189336:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574421.189341:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574421.189354:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5569 00000100:00080000:3.0:1788574421.189369:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574421.189378:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574421.189425:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488770048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574421.189459:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574421.189463:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574421.189464:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574421.189472:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5569 00000100:00080000:3.0:1788574421.189476:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574421.189481:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574421.189493:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574421.189506:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488770048:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574421.189510:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488770176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574421.189525:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574421.189528:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574421.189530:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574421.189535:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5569 00000100:00080000:3.0:1788574421.189540:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574421.189544:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574421.189550:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574421.189563:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488770304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574421.189573:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574421.189577:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574421.189580:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574421.189584:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5569 00000100:00080000:3.0:1788574421.189589:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574421.189593:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574421.189599:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788574421.189605:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574421.189616:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488770432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574421.189626:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574421.189630:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574421.189632:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574421.189636:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5569 00000100:00080000:3.0:1788574421.189649:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574421.189654:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574421.189660:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574421.189668:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574421/real 1788574421] req@ffff94d408c56a00 x1875450488770048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574421.189676:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488770560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574421.189722:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488770048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574421.189737:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574421.189743:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574421.189748:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488770048:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788574421.189795:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488770176:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574421.189827:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574421.189843:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574421/real 1788574421] req@ffff94d408c55880 x1875450488770176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574421.189856:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488770176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574421.189864:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574421.189867:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574421.189870:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488770176:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574421.189876:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488770304:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574421.189898:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574421.189910:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574421/real 1788574421] req@ffff94d408c54380 x1875450488770304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574421.189922:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488770304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574421.189936:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574421.189941:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574421.189944:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488770304:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574421.189951:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488770432:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574421.189975:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574421.190020:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574421/real 1788574421] req@ffff94d408c57800 x1875450488770432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574421.190041:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488770432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574421.190050:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574421.190053:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574421.190056:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488770432:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574421.190063:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488770560:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574421.190088:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574421.190101:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574421/real 1788574421] req@ffff94d408c55180 x1875450488770560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574421.190112:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488770560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574421.190120:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574421.190124:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574421.190128:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488770560:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574421.701274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574422.149356:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574422.725301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574423.173337:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574423.749324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574424.197396:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574424.773350:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574425.221364:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574425.797365:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574426.245328:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574426.309351:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574426.309357:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574426.309361:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574426.309369:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5574 00000100:00080000:3.0:1788574426.309378:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574426.309385:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574426.309421:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488770688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574426.309453:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574426.309457:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574426.309459:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574426.309465:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5574 00000100:00080000:3.0:1788574426.309469:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574426.309475:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574426.309484:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574426.309499:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488770688:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574426.309501:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488770816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574426.309513:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574426.309517:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574426.309520:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574426.309524:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5574 00000100:00080000:3.0:1788574426.309528:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574426.309532:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574426.309537:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574426.309546:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488770944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574426.309554:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788574426.309558:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574426.309558:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574426.309560:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574426.309563:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5574 00000100:00080000:3.0:1788574426.309566:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574426.309570:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574426.309574:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574426.309590:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574426/real 1788574426] req@ffff94d408c57100 x1875450488770688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574426.309608:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488770688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574426.309619:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574426.309623:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574426.309628:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488770688:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574426.309660:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488770816:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574426.309664:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488771072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574426.309674:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574426.309678:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574426.309680:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788574426.309683:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574426.309683:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5574 00000100:00080000:3.0:1788574426.309687:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574426.309691:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574426.309696:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574426/real 1788574426] req@ffff94d408c55c00 x1875450488770816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788574426.309696:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574426.309705:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488771200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574426.309709:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488770816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574426.309718:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574426.309721:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574426.309725:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488770816:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574426.309731:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488770944:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574426.309752:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574426.309768:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574426/real 1788574426] req@ffff94d408c54e00 x1875450488770944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574426.309782:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488770944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574426.309791:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574426.309794:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574426.309796:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488770944:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574426.309810:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488771072:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574426.309828:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574426.309842:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574426/real 1788574426] req@ffff94d408c56d80 x1875450488771072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574426.309855:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488771072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574426.309864:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574426.309867:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574426.309871:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488771072:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574426.309878:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488771200:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574426.309898:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574426.309910:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574426/real 1788574426] req@ffff94d408c54a80 x1875450488771200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574426.309921:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488771200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574426.309930:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574426.309934:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574426.309937:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488771200:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574426.821647:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574427.269333:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574427.845409:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574428.293349:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574428.869347:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574429.317399:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574429.893976:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574430.341340:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574430.917315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574431.365414:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574431.429341:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574431.429354:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574431.429360:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574431.429373:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5579 00000100:00080000:3.0:1788574431.429387:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574431.429396:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574431.429447:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488771328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574431.429482:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574431.429486:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574431.429488:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574431.429492:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5579 00000100:00080000:3.0:1788574431.429496:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574431.429500:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574431.429511:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574431.429523:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488771456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574431.429533:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574431.429538:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574431.429541:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574431.429545:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5579 00000100:00080000:3.0:1788574431.429549:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574431.429553:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574431.429559:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574431.429566:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488771328:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574431.429574:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488771584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574431.429586:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574431.429590:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574431.429592:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574431.429596:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5579 00000100:00080000:3.0:1788574431.429600:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574431.429604:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574431.429610:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574431.429621:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488771712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574431.429631:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574431.429634:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574431.429636:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574431.429640:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5579 00000100:00080000:3.0:1788574431.429644:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788574431.429645:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574431.429653:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574431.429660:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574431.429673:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488771840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574431.429691:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574431/real 1788574431] req@ffff94d408c57b80 x1875450488771328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574431.429711:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488771328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574431.429725:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574431.429729:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574431.429734:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488771328:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574431.429742:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488771456:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574431.429769:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574431.429782:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574431/real 1788574431] req@ffff94d408c56680 x1875450488771456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574431.429792:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488771456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574431.429798:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574431.429800:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574431.429802:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488771456:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574431.429825:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488771584:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574431.429839:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574431.429845:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574431/real 1788574431] req@ffff94d408c54a80 x1875450488771584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574431.429853:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488771584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574431.429859:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574431.429861:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574431.429862:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488771584:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574431.429867:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488771712:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574431.429878:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574431.429899:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574431/real 1788574431] req@ffff94d408c56d80 x1875450488771712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574431.429907:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488771712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574431.429913:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574431.429915:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574431.429917:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488771712:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574431.429921:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488771840:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574431.429933:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574431.429939:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574431/real 1788574431] req@ffff94d408c54e00 x1875450488771840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574431.429946:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488771840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574431.429951:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574431.429953:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574431.429955:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488771840:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574431.941309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574432.389354:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574432.965332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574433.413408:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574433.989268:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574434.437429:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574435.013301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574435.461346:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574436.037277:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574436.485364:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574436.549309:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574436.549320:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574436.549325:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574436.549336:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5584 00000100:00080000:3.0:1788574436.549347:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574436.549356:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574436.549394:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488771968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574436.549422:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574436.549426:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574436.549428:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574436.549433:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5584 00000100:00080000:3.0:1788574436.549436:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574436.549440:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574436.549450:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574436.549466:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488772096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574436.549478:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488771968:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574436.549481:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574436.549489:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574436.549491:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574436.549496:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5584 00000100:00080000:3.0:1788574436.549499:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574436.549504:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574436.549510:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574436.549524:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488772224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574436.549541:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788574436.549546:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574436.549550:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574436.549554:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574436.549560:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5584 00000100:00080000:3.0:1788574436.549567:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574436.549572:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574436.549578:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574436.549582:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574436/real 1788574436] req@ffff94d408c55c00 x1875450488771968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574436.549595:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488772352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574436.549600:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488771968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574436.549622:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574436.549623:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574436.549629:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:3.0:1788574436.549631:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1788574436.549635:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488771968:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574436.549636:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574436.549646:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5584 00000100:00080000:3.0:1788574436.549650:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574436.549655:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574436.549660:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574436.549666:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488772096:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574436.549674:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488772480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574436.549698:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574436.549712:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574436/real 1788574436] req@ffff94d408c57100 x1875450488772096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574436.549721:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488772096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574436.549727:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574436.549730:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574436.549732:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488772096:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574436.549736:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488772224:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574436.549749:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574436.549759:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574436/real 1788574436] req@ffff94d408c55180 x1875450488772224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574436.549772:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488772224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574436.549803:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574436.549807:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574436.549810:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488772224:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574436.549815:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488772352:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574436.549835:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574436.549846:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574436/real 1788574436] req@ffff94d408c57800 x1875450488772352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574436.549859:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488772352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574436.549868:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574436.549872:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574436.549875:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488772352:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574436.549889:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488772480:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574436.549904:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574436.549913:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574436/real 1788574436] req@ffff94d408c54380 x1875450488772480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574436.549920:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488772480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574436.549926:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574436.549928:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574436.549930:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488772480:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574437.061257:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574437.509345:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574438.085266:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574438.533368:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574439.109274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574439.557347:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574440.133317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574440.581348:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574441.157305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574441.605379:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574441.669271:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574441.669279:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574441.669282:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574441.669290:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5589 00000100:00080000:3.0:1788574441.669299:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574441.669305:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574441.669337:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488772608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574441.669367:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574441.669371:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574441.669373:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574441.669377:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5589 00000100:00080000:3.0:1788574441.669381:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574441.669386:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574441.669395:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574441.669408:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488772736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574441.669409:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488772608:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574441.669419:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574441.669422:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574441.669424:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574441.669428:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5589 00000100:00080000:3.0:1788574441.669431:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574441.669434:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574441.669439:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574441.669448:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488772864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574441.669456:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574441.669458:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574441.669460:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574441.669463:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5589 00000100:00080000:3.0:1788574441.669466:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788574441.669468:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574441.669470:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574441.669474:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574441.669482:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488772992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574441.669491:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574441.669494:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574441.669496:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574441.669499:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5589 00000100:00100000:0.0:1788574441.669501:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574441/real 1788574441] req@ffff94d408c55880 x1875450488772608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574441.669502:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574441.669506:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574441.669519:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488772608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788574441.669519:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574441.669529:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488773120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574441.669530:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574441.669535:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574441.669538:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488772608:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574441.669560:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488772736:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574441.669585:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574441.669596:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574441/real 1788574441] req@ffff94d408c56a00 x1875450488772736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574441.669605:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488772736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574441.669611:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574441.669613:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574441.669615:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488772736:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574441.669619:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488772864:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574441.669631:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574441.669638:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574441/real 1788574441] req@ffff94d408c54000 x1875450488772864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574441.669645:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488772864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574441.669651:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574441.669652:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574441.669654:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488772864:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574441.669658:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488772992:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574441.669668:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574441.669674:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574441/real 1788574441] req@ffff94d408c56300 x1875450488772992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574441.669682:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488772992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574441.669687:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574441.669688:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574441.669690:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488772992:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574441.669693:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488773120:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574441.669702:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574441.669708:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574441/real 1788574441] req@ffff94d408c55f80 x1875450488773120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574441.669715:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488773120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574441.669721:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574441.669723:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574441.669725:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488773120:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574442.181335:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574442.629355:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574443.205372:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574443.653400:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574444.229329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574444.677366:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574445.253339:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574445.701396:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574446.277360:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574446.725369:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574446.789372:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574446.789382:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574446.789387:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574446.789398:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5595 00000100:00080000:0.0:1788574446.789412:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574446.789422:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574446.789476:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488773248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574446.789514:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574446.789518:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574446.789519:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574446.789522:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5595 00000100:00080000:0.0:1788574446.789526:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574446.789529:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574446.789538:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574446.789549:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488773376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574446.789558:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574446.789561:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574446.789563:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574446.789570:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5595 00000100:00080000:0.0:1788574446.789572:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574446.789576:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574446.789580:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488773248:192.168.202.125@tcp:250:kworker.0 00000008:00100000:0.0:1788574446.789581:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574446.789595:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488773504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574446.789603:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574446.789606:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574446.789607:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574446.789610:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5595 00000100:00080000:0.0:1788574446.789613:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574446.789615:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574446.789618:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574446.789627:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488773632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574446.789632:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574446.789635:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574446.789637:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574446.789639:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5595 00000100:00080000:0.0:1788574446.789649:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574446.789652:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574446.789655:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788574446.789663:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574446.789665:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488773760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574446.789706:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574446/real 1788574446] req@ffff94d408d3e680 x1875450488773248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574446.789721:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488773248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574446.789729:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574446.789733:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574446.789736:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488773248:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788574446.789742:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488773376:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574446.789765:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574446.789781:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574446/real 1788574446] req@ffff94d408d3ea00 x1875450488773376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574446.789796:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488773376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574446.789805:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574446.789809:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574446.789813:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488773376:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574446.789849:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488773504:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574446.789868:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574446.789880:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574446/real 1788574446] req@ffff94d408d3df80 x1875450488773504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574446.789893:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488773504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574446.789903:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574446.789906:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574446.789910:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488773504:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574446.789917:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488773632:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574446.789938:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574446.789947:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574446/real 1788574446] req@ffff94d408d3d180 x1875450488773632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574446.789961:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488773632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574446.789966:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574446.789968:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574446.789970:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488773632:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574446.789974:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488773760:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574446.789986:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574446.789991:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574446/real 1788574446] req@ffff94d408d3c000 x1875450488773760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574446.789999:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488773760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574446.790004:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574446.790006:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574446.790008:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488773760:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574447.301361:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574447.749414:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574448.325366:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574448.773473:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574449.349324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574449.797420:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574450.373321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574450.821473:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574451.397302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574451.845401:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574451.909295:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574451.909304:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574451.909308:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574451.909336:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5600 00000100:00080000:0.0:1788574451.909347:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574451.909353:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574451.909387:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488773888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574451.909410:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574451.909413:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574451.909415:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574451.909418:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5600 00000100:00080000:0.0:1788574451.909420:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574451.909422:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574451.909429:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574451.909438:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488774016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574451.909446:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574451.909448:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574451.909449:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788574451.909454:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488773888:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788574451.909455:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5600 00000100:00080000:0.0:1788574451.909459:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574451.909464:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574451.909467:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574451.909475:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488774144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574451.909482:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574451.909485:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574451.909486:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574451.909488:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5600 00000100:00080000:0.0:1788574451.909501:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574451.909504:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574451.909507:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788574451.909513:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574451.909517:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488774272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574451.909525:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574451.909527:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574451.909529:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574451.909532:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5600 00000100:00080000:0.0:1788574451.909534:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574451.909537:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574451.909540:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574451.909548:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488774400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574451.909552:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574451/real 1788574451] req@ffff94d408d3c380 x1875450488773888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574451.909575:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488773888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574451.909585:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574451.909589:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574451.909593:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488773888:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788574451.909617:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488774016:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574451.909642:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574451.909656:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574451/real 1788574451] req@ffff94d408d3f480 x1875450488774016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574451.909665:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488774016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574451.909671:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574451.909673:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574451.909675:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488774016:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574451.909679:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488774144:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574451.909693:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574451.909703:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574451/real 1788574451] req@ffff94d408d3c700 x1875450488774144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574451.909711:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488774144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574451.909721:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574451.909723:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574451.909725:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488774144:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574451.909729:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488774272:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574451.909739:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574451.909744:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574451/real 1788574451] req@ffff94d408d3e300 x1875450488774272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574451.909751:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488774272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574451.909756:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574451.909758:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574451.909760:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488774272:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574451.909764:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488774400:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574451.909776:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574451.909782:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574451/real 1788574451] req@ffff94d408d3dc00 x1875450488774400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574451.909790:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488774400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574451.909794:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574451.909796:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574451.909799:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488774400:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574452.421306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574452.869469:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574453.445279:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574453.893398:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574454.469298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574454.917426:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574455.493268:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574455.941400:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574456.517278:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574456.965326:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574457.029299:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574457.029307:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574457.029311:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574457.029319:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5605 00000100:00080000:0.0:1788574457.029328:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574457.029334:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574457.029367:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488774528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574457.029405:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574457.029408:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574457.029409:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574457.029412:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5605 00000100:00080000:0.0:1788574457.029415:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574457.029418:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574457.029426:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574457.029436:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ed80 x1875450488774656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574457.029444:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574457.029446:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574457.029448:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788574457.029452:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488774528:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788574457.029453:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5605 00000100:00080000:0.0:1788574457.029458:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574457.029462:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574457.029465:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574457.029475:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488774784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574457.029483:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574457.029485:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574457.029487:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574457.029489:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5605 00000100:00080000:0.0:1788574457.029500:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574457.029503:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574457.029506:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788574457.029514:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574457.029516:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488774912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574457.029523:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574457.029526:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574457.029528:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574457.029530:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5605 00000100:00080000:0.0:1788574457.029533:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574457.029535:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574457.029539:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574457.029549:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488775040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574457.029549:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574457/real 1788574457] req@ffff94d408d3d880 x1875450488774528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574457.029591:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488774528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574457.029600:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574457.029602:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574457.029606:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488774528:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788574457.029625:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488774656:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574457.029640:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574457.029648:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574457/real 1788574457] req@ffff94d408d3ed80 x1875450488774656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574457.029656:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ed80 x1875450488774656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574457.029661:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574457.029664:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574457.029666:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488774656:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574457.029673:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488774784:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574457.029692:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574457.029704:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574457/real 1788574457] req@ffff94d408d3dc00 x1875450488774784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574457.029717:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488774784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574457.029726:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574457.029730:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574457.029734:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488774784:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574457.029740:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488774912:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574457.029761:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574457.029773:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574457/real 1788574457] req@ffff94d408d3e300 x1875450488774912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574457.029786:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488774912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574457.029795:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574457.029798:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574457.029802:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488774912:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574457.029808:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488775040:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574457.029829:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574457.029844:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574457/real 1788574457] req@ffff94d408d3c700 x1875450488775040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574457.029853:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488775040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574457.029858:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574457.029861:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574457.029864:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488775040:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574457.541343:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574457.989359:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574458.565285:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574459.013349:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574459.589383:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574460.037455:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574460.613286:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574461.061434:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574461.637349:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574462.085416:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574462.149313:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574462.149325:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574462.149330:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574462.149342:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5610 00000100:00080000:0.0:1788574462.149355:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574462.149369:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574462.149411:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488775168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574462.149456:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574462.149459:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574462.149461:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574462.149467:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5610 00000100:00080000:0.0:1788574462.149471:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574462.149476:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574462.149489:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574462.149507:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488775296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574462.149509:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488775168:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788574462.149545:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574462.149552:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574462.149555:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574462.149561:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5610 00000100:00080000:0.0:1788574462.149567:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574462.149570:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574462.149574:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574462.149589:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488775424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574462.149596:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574462.149600:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574462.149601:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574462.149606:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5610 00000100:00080000:0.0:1788574462.149615:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788574462.149615:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788574462.149630:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574462.149636:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574462.149648:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488775552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574462.149656:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574462.149659:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574462.149661:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574462.149665:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5610 00000100:00080000:0.0:1788574462.149668:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574462.149670:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574462.149674:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574462.149678:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574462/real 1788574462] req@ffff94d408d3f480 x1875450488775168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574462.149685:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488775680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574462.149748:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488775168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574462.149763:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574462.149768:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574462.149772:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488775168:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574462.149799:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488775296:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574462.149823:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574462.149836:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574462/real 1788574462] req@ffff94d408d3c380 x1875450488775296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574462.149847:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488775296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574462.149853:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574462.149858:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574462.149862:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488775296:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574462.149866:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488775424:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574462.149881:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574462.149891:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574462/real 1788574462] req@ffff94d408d3c000 x1875450488775424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574462.149912:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488775424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574462.149923:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574462.149929:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574462.149932:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488775424:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574462.149944:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488775552:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574462.149977:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574462.149999:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574462/real 1788574462] req@ffff94d408d3d180 x1875450488775552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574462.150020:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488775552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574462.150028:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574462.150031:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574462.150035:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488775552:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574462.150040:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488775680:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574462.150073:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574462.150084:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574462/real 1788574462] req@ffff94d408d3df80 x1875450488775680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574462.150095:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488775680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574462.150108:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574462.150111:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574462.150114:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488775680:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574462.661354:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574463.109381:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574463.685316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574464.133373:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574464.709305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574465.157357:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574465.733319:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574466.181417:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574466.757351:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574467.205433:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574467.269331:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574467.269346:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574467.269351:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574467.269365:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5615 00000100:00080000:2.0:1788574467.269378:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574467.269388:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574467.269442:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60700 x1875450488775808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574467.269481:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574467.269485:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1788574467.269560:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488775808:192.168.202.125@tcp:250:kworker.0 00000800:00000100:1.0:1788574467.269625:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574467.269656:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574467/real 1788574467] req@ffff94d408c60700 x1875450488775808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574467.269656:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574467.269672:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5615 00000100:00080000:2.0:1788574467.269689:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574467.269694:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60700 x1875450488775808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574467.269702:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00080000:1.0:1788574467.269713:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574467.269719:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000008:00100000:2.0:1788574467.269721:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574467.269722:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488775808:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574467.269750:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60380 x1875450488775936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574467.269772:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574467.269777:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574467.269781:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788574467.269789:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488775936:192.168.202.125@tcp:38:kworker.0 00000100:00080000:2.0:1788574467.269790:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5615 00000100:00080000:2.0:1788574467.269803:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574467.269811:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574467.269819:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788574467.269822:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574467.269839:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574467/real 1788574467] req@ffff94d408c60380 x1875450488775936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574467.269849:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60380 x1875450488775936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574467.269859:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574467.269864:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574467.269864:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62300 x1875450488776064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574467.269875:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488775936:192.168.202.125@tcp:38:kworker.0 00000100:00080000:2.0:1788574467.269907:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574467.269911:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574467.269914:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574467.269920:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5615 00000100:00080000:2.0:1788574467.269925:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574467.269926:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488776064:192.168.202.125@tcp:38:kworker.0 00000100:00080000:2.0:1788574467.269935:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574467.269942:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788574467.269949:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574467.269961:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60380 x1875450488776192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574467.269967:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574467/real 1788574467] req@ffff94d408c62300 x1875450488776064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574467.270005:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574467.270014:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1788574467.270015:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62300 x1875450488776064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574467.270021:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574467.270032:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5615 00000100:00080000:2.0:1788574467.270034:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574467.270037:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574467.270053:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788574467.270054:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574467.270065:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574467.270073:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488776064:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574467.270073:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60700 x1875450488776320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574467.270108:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488776192:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574467.270130:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574467.270138:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574467/real 1788574467] req@ffff94d408c60380 x1875450488776192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574467.270151:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60380 x1875450488776192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574467.270156:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574467.270158:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574467.270160:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488776192:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574467.270164:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488776320:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574467.270174:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574467.270179:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574467/real 1788574467] req@ffff94d408c60700 x1875450488776320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574467.270186:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60700 x1875450488776320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574467.270191:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574467.270193:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574467.270196:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488776320:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574467.781332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574468.229403:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574468.805287:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574469.253409:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574469.829307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574470.277437:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574470.853320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574471.301334:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574471.877349:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574472.325411:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574472.389347:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574472.389358:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574472.389362:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574472.389373:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5620 00000100:00080000:1.0:1788574472.389386:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574472.389395:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574472.389456:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488776448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574472.389494:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574472.389498:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574472.389499:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574472.389503:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5620 00000100:00080000:1.0:1788574472.389506:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574472.389510:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574472.389520:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574472.389534:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488776576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574472.389543:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574472.389562:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574472.389564:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574472.389568:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5620 00000100:00080000:1.0:1788574472.389570:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574472.389573:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574472.389578:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574472.389583:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488776448:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788574472.389592:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488776704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574472.389605:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574472.389608:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574472.389611:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574472.389616:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5620 00000100:00080000:1.0:1788574472.389620:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574472.389624:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574472.389631:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574472.389644:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488776832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574472.389652:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574472.389655:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574472.389657:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574472.389660:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5620 00000100:00080000:1.0:1788574472.389663:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574472.389666:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574472.389670:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574472.389675:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574472.389686:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488776960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574472.389725:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574472/real 1788574472] req@ffff94d4049bfb80 x1875450488776448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574472.389748:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488776448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574472.389762:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574472.389766:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574472.389770:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488776448:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574472.389801:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488776576:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574472.389828:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574472.389842:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574472/real 1788574472] req@ffff94d4049bc000 x1875450488776576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574472.389856:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488776576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574472.389866:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574472.389869:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574472.389872:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488776576:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574472.389880:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488776704:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574472.389905:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574472.389918:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574472/real 1788574472] req@ffff94d4049bc700 x1875450488776704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574472.389933:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488776704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574472.389941:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574472.389944:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574472.389948:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488776704:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574472.389955:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488776832:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574472.389978:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574472.389989:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574472/real 1788574472] req@ffff94d4049bed80 x1875450488776832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574472.390002:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488776832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574472.390011:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574472.390015:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574472.390019:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488776832:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574472.390025:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488776960:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574472.390046:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574472.390059:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574472/real 1788574472] req@ffff94d4049bf800 x1875450488776960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574472.390079:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488776960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574472.390088:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574472.390092:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574472.390095:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488776960:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574472.901274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574473.349424:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574473.925298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574474.373380:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574474.949265:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574475.397366:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574475.973326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574476.421362:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574476.997249:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574477.445361:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574477.509354:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574477.509362:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574477.509366:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574477.509375:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5625 00000100:00080000:3.0:1788574477.509384:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574477.509389:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574477.509420:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488777088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574477.509442:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574477.509444:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574477.509445:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574477.509448:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5625 00000100:00080000:3.0:1788574477.509450:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574477.509453:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574477.509460:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574477.509469:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488777216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574477.509477:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574477.509480:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574477.509483:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574477.509484:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488777088:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574477.509487:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5625 00000100:00080000:3.0:1788574477.509491:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574477.509496:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574477.509500:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574477.509508:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488777344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574477.509514:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574477.509516:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574477.509517:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574477.509520:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5625 00000100:00080000:3.0:1788574477.509522:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574477.509524:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574477.509527:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574477.509534:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488777472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574477.509540:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574477.509542:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574477.509543:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574477.509545:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5625 00000100:00080000:3.0:1788574477.509558:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574477.509560:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574477.509564:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574477.509571:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574477.509573:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488777600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574477.509627:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574477/real 1788574477] req@ffff94d408c57480 x1875450488777088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574477.509648:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488777088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574477.509659:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574477.509663:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574477.509667:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488777088:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574477.509687:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488777216:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574477.509705:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574477.509714:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574477/real 1788574477] req@ffff94d408c54700 x1875450488777216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574477.509722:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488777216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574477.509732:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574477.509734:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574477.509736:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488777216:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574477.509740:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488777344:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574477.509750:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574477.509756:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574477/real 1788574477] req@ffff94d408c55f80 x1875450488777344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574477.509763:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488777344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574477.509768:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574477.509770:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574477.509772:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488777344:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574477.509775:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488777472:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574477.509786:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574477.509792:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574477/real 1788574477] req@ffff94d408c56300 x1875450488777472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574477.509799:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488777472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574477.509804:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574477.509805:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574477.509808:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488777472:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574477.509811:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488777600:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574477.509821:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574477.509826:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574477/real 1788574477] req@ffff94d408c54000 x1875450488777600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574477.509833:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488777600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574477.509838:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574477.509840:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574477.509842:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488777600:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574478.021262:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574478.469354:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574479.045320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574479.493354:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574480.069306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574480.517338:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574481.093288:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574481.541391:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574482.117268:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574482.565403:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574482.629274:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574482.629284:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574482.629287:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574482.629296:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5630 00000100:00080000:3.0:1788574482.629305:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574482.629311:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574482.629344:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488777728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574482.629367:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574482.629370:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574482.629371:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574482.629374:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5630 00000100:00080000:3.0:1788574482.629376:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574482.629379:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574482.629387:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574482.629397:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488777856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574482.629405:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574482.629408:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788574482.629412:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488777728:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574482.629418:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574482.629423:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5630 00000100:00080000:3.0:1788574482.629428:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574482.629432:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574482.629435:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574482.629445:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488777984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574482.629452:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574482.629455:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574482.629456:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574482.629459:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5630 00000100:00080000:3.0:1788574482.629473:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574482.629475:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574482.629479:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574482.629487:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488778112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574482.629494:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574482.629497:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574482.629499:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574482.629501:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5630 00000100:00080000:3.0:1788574482.629503:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574482.629506:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574482.629510:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574482.629512:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488777856:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574482.629519:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488778240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574482.629542:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488777984:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574482.629561:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488778112:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574482.629578:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488778240:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574483.141326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574483.589385:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574484.165316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574484.613352:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574485.189315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574485.637378:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788574485.701435:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:1.0:1788574485.701479:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788574485.701502:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788574485.701539:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788574485.701549:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788574485.701560:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788574485.701571:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:2.0:1788574485.701593:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574482/real 1788574485] req@ffff94d408c55180 x1875450488778240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574485.701639:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488778240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574485.701668:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574485.701674:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574485.701679:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488778240:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574485.701687:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574482/real 1788574485] req@ffff94d408c55880 x1875450488777856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574485.701713:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488777856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574485.701722:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574485.701726:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574485.701730:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488777856:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574485.701738:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574482/real 1788574485] req@ffff94d408c54380 x1875450488777984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574485.701749:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488777984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574485.701758:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574485.701762:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574485.701766:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488777984:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574485.701773:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574482/real 1788574485] req@ffff94d408c57800 x1875450488778112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574485.701786:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488778112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574485.701795:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574485.701798:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574485.701802:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488778112:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574485.701809:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574482/real 1788574485] req@ffff94d408c56a00 x1875450488777728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574485.701822:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488777728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574485.701831:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574485.701835:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574485.701839:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488777728:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574486.213294:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574486.661374:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574487.237310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574487.685391:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574487.749445:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574487.749456:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574487.749461:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574487.749475:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5636 00000100:00080000:3.0:1788574487.749486:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574487.749493:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574487.749528:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488778368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574487.749564:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574487.749568:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574487.749570:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574487.749575:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5636 00000100:00080000:3.0:1788574487.749578:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574487.749583:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574487.749593:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574487.749610:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488778496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574487.749620:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488778368:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574487.749624:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574487.749632:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574487.749634:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574487.749645:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5636 00000100:00080000:3.0:1788574487.749649:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574487.749653:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574487.749659:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574487.749672:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488778624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574487.749682:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574487.749686:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574487.749689:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574487.749693:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5636 00000100:00080000:3.0:1788574487.749706:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574487.749711:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:2.0:1788574487.749731:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:3.0:1788574487.749743:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574487.749769:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488778752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574487.749777:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574487/real 1788574487] req@ffff94d408c57100 x1875450488778368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574487.749783:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574487.749791:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574487.749795:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574487.749802:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5636 00000100:00080000:3.0:1788574487.749806:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788574487.749810:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488778368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574487.749815:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574487.749824:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788574487.749830:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788574487.749838:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488778880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574487.749839:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574487.749847:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488778368:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574487.749884:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488778496:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574487.749913:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574487.749928:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574487/real 1788574487] req@ffff94d408c55c00 x1875450488778496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574487.749942:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488778496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574487.749950:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574487.749954:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574487.749958:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488778496:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574487.749965:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488778624:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574487.749987:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574487.750001:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574487/real 1788574487] req@ffff94d408c54e00 x1875450488778624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574487.750014:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488778624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574487.750023:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574487.750027:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574487.750030:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488778624:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574487.750037:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488778752:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574487.750058:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574487.750070:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574487/real 1788574487] req@ffff94d408c56d80 x1875450488778752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574487.750091:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488778752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574487.750100:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574487.750103:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574487.750106:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488778752:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574487.750112:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488778880:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574487.750132:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574487.750144:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574487/real 1788574487] req@ffff94d408c54a80 x1875450488778880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574487.750157:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488778880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574487.750165:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574487.750169:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574487.750172:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488778880:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574488.261318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574488.709338:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574489.285310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574489.733360:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574490.309278:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574490.757330:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574491.333745:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574491.781364:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574492.357378:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574492.805381:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574492.869359:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574492.869369:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574492.869374:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574492.869386:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5641 00000100:00080000:0.0:1788574492.869400:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574492.869411:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574492.869468:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488779008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574492.869497:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574492.869500:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574492.869502:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574492.869505:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5641 00000100:00080000:0.0:1788574492.869508:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574492.869511:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574492.869526:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574492.869539:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488779136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574492.869547:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574492.869550:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574492.869552:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574492.869555:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5641 00000100:00080000:0.0:1788574492.869558:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574492.869562:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574492.869565:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574492.869579:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488779264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574492.869582:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488779008:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788574492.869606:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574492.869611:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574492.869614:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574492.869621:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5641 00000100:00080000:0.0:1788574492.869628:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574492.869634:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574492.869641:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574492.869657:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488779392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574492.869669:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788574492.869678:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574492.869681:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574492.869683:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574492.869690:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5641 00000100:00080000:0.0:1788574492.869695:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574492.869700:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574492.869705:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574492.869713:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574492/real 1788574492] req@ffff94d408d3ea00 x1875450488779008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574492.869719:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488779520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574492.869799:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488779008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574492.869818:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574492.869823:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574492.869828:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488779008:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574492.869837:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488779136:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574492.869864:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574492.869879:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574492/real 1788574492] req@ffff94d408d3e680 x1875450488779136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574492.869894:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488779136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574492.869903:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574492.869907:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574492.869910:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488779136:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574492.869940:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488779264:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574492.869963:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574492.869975:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574492/real 1788574492] req@ffff94d408d3d500 x1875450488779264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574492.869988:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488779264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574492.869998:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574492.870001:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574492.870005:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488779264:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574492.870013:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488779392:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574492.870030:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574492.870038:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574492/real 1788574492] req@ffff94d408d3ca80 x1875450488779392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574492.870045:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488779392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574492.870051:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574492.870053:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574492.870055:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488779392:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574492.870059:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488779520:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574492.870068:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574492.870074:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574492/real 1788574492] req@ffff94d408d3ce00 x1875450488779520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574492.870081:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488779520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574492.870086:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574492.870088:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574492.870091:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488779520:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574493.381765:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574493.829359:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574494.405351:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574494.853386:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574495.429326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574495.877373:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574496.453351:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574496.901370:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574497.477301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574497.925377:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574497.989299:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574497.989307:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574497.989311:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574497.989319:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5646 00000100:00080000:0.0:1788574497.989328:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574497.989334:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574497.989366:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f100 x1875450488779648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574497.989390:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574497.989395:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574497.989397:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574497.989400:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5646 00000100:00080000:0.0:1788574497.989403:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574497.989405:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574497.989413:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574497.989424:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f800 x1875450488779776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574497.989433:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574497.989440:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488779648:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788574497.989443:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574497.989447:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574497.989452:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5646 00000100:00080000:0.0:1788574497.989455:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574497.989458:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574497.989462:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574497.989476:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488779904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574497.989483:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574497.989486:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574497.989488:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574497.989491:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5646 00000100:00080000:0.0:1788574497.989494:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574497.989497:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574497.989502:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788574497.989512:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574497.989513:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488780032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574497.989522:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574497.989524:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574497.989525:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574497.989530:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5646 00000100:00080000:0.0:1788574497.989534:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574497.989539:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574497.989544:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574497.989554:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574497/real 1788574497] req@ffff94d408d3f100 x1875450488779648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574497.989559:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488780160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574497.989591:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f100 x1875450488779648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574497.989604:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574497.989608:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574497.989614:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488779648:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788574497.989642:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488779776:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574497.989673:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574497.989684:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574497/real 1788574497] req@ffff94d408d3f800 x1875450488779776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574497.989697:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f800 x1875450488779776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574497.989705:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574497.989708:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574497.989712:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488779776:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574497.989718:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488779904:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574497.989740:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574497.989752:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574497/real 1788574497] req@ffff94d408d3ce00 x1875450488779904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574497.989764:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488779904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574497.989772:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574497.989776:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574497.989779:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488779904:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574497.989786:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488780032:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574497.989823:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574497.989837:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574497/real 1788574497] req@ffff94d408d3ca80 x1875450488780032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574497.989851:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488780032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574497.989861:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574497.989864:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574497.989869:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488780032:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574497.989876:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488780160:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574497.989902:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574497.989915:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574497/real 1788574497] req@ffff94d408d3d500 x1875450488780160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574497.989928:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488780160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574497.989937:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574497.989941:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574497.989945:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488780160:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574498.501301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574498.949413:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574499.525273:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574499.973340:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574500.549316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574500.997441:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574501.573295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574502.021370:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574502.597293:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574503.045387:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574503.109353:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574503.109369:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574503.109374:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574503.109387:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5651 00000100:00080000:1.0:1788574503.109402:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574503.109413:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574503.109465:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488780288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574503.109505:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574503.109510:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574503.109512:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574503.109518:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5651 00000100:00080000:1.0:1788574503.109523:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574503.109529:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574503.109541:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574503.109557:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488780288:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788574503.109563:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488780416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574503.109580:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574503.109584:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574503.109587:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574503.109592:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5651 00000100:00080000:1.0:1788574503.109597:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574503.109602:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574503.109608:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788574503.109624:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574503.109626:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488780544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574503.109641:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574503.109646:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574503.109648:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574503.109656:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5651 00000100:00080000:1.0:1788574503.109662:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788574503.109663:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574503/real 1788574503] req@ffff94d4049be680 x1875450488780288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574503.109676:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574503.109687:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574503.109687:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488780288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574503.109706:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574503.109714:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574503.109716:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488780672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574503.109719:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488780288:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788574503.109744:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574503.109749:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574503.109752:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574503.109754:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488780416:192.168.202.125@tcp:38:kworker.0 00000100:00080000:1.0:1788574503.109775:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5651 00000800:00000100:2.0:1788574503.109781:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788574503.109791:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788574503.109797:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574503/real 1788574503] req@ffff94d4049bf100 x1875450488780416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574503.109806:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574503.109838:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574503.109845:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488780416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574503.109855:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574503.109868:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574503.109875:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488780416:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574503.109876:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488780800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574503.109898:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488780544:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574503.109924:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574503.109942:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574503/real 1788574503] req@ffff94d4049bca80 x1875450488780544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574503.109957:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488780544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574503.109973:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574503.109977:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574503.109980:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488780544:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574503.109985:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488780672:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574503.110002:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574503.110018:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574503/real 1788574503] req@ffff94d4049bd500 x1875450488780672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574503.110026:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488780672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574503.110032:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574503.110034:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574503.110036:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488780672:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574503.110049:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488780800:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574503.110061:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574503.110068:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574503/real 1788574503] req@ffff94d4049bdc00 x1875450488780800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574503.110075:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488780800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574503.110080:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574503.110082:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574503.110085:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488780800:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574503.621291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574504.069456:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574504.645280:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574505.093336:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574505.669300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574506.117343:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574506.693371:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574507.141383:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574507.717294:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574508.165384:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574508.229340:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574508.229355:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574508.229361:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574508.229373:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5656 00000100:00080000:1.0:1788574508.229387:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574508.229397:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574508.229448:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488780928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574508.229486:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574508.229490:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574508.229493:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574508.229499:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5656 00000100:00080000:1.0:1788574508.229503:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574508.229509:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574508.229522:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574508.229543:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd880 x1875450488781056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574508.229543:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488780928:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788574508.229572:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574508.229578:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574508.229582:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574508.229589:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5656 00000100:00080000:1.0:1788574508.229594:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574508.229600:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574508.229606:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574508.229626:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488781184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574508.229627:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788574508.229645:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574508.229650:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574508.229652:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574508.229658:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5656 00000100:00080000:1.0:1788574508.229664:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574508.229670:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574508.229677:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574508.229682:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574508/real 1788574508] req@ffff94d4049be300 x1875450488780928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574508.229697:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488781312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574508.229703:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488780928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574508.229744:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574508.229748:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574508.229749:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:1.0:1788574508.229762:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1788574508.229762:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488780928:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788574508.229773:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574508.229781:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5656 00000100:00080000:1.0:1788574508.229787:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574508.229792:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574508.229799:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574508.229805:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488781056:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574508.229819:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488781440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574508.229846:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574508.229877:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574508/real 1788574508] req@ffff94d4049bd880 x1875450488781056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574508.229893:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd880 x1875450488781056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574508.229903:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574508.229907:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574508.229911:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488781056:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574508.229917:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488781184:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574508.229938:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574508.229950:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574508/real 1788574508] req@ffff94d4049bdc00 x1875450488781184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574508.229963:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488781184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574508.229973:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574508.229976:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574508.229980:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488781184:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574508.229986:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488781312:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574508.230003:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574508.230011:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574508/real 1788574508] req@ffff94d4049bd500 x1875450488781312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574508.230019:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488781312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574508.230024:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574508.230026:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574508.230028:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488781312:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574508.230040:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488781440:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574508.230050:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574508.230056:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574508/real 1788574508] req@ffff94d4049bca80 x1875450488781440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574508.230063:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488781440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574508.230068:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574508.230070:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574508.230072:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488781440:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574508.741305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574509.189395:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574509.765309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574510.213401:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574510.789332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574511.237406:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574511.813267:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574512.261361:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574512.837323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574513.285355:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574513.349308:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574513.349316:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574513.349319:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574513.349328:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5661 00000100:00080000:1.0:1788574513.349337:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574513.349343:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574513.349375:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488781568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574513.349399:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574513.349402:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574513.349403:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574513.349406:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5661 00000100:00080000:1.0:1788574513.349408:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574513.349410:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574513.349417:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574513.349425:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488781696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574513.349432:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574513.349435:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574513.349436:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574513.349439:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5661 00000100:00080000:1.0:1788574513.349441:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574513.349443:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574513.349447:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574513.349455:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488781824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574513.349455:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488781568:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788574513.349469:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574513.349471:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574513.349472:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574513.349475:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5661 00000100:00080000:1.0:1788574513.349477:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574513.349480:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574513.349483:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574513.349492:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488781952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574513.349498:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574513.349500:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574513.349501:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574513.349503:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5661 00000100:00080000:1.0:1788574513.349505:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574513.349508:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574513.349511:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574513.349520:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488782080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574513.349520:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574513.349558:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574513/real 1788574513] req@ffff94d4049bf100 x1875450488781568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574513.349572:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488781568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574513.349579:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574513.349582:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574513.349585:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488781568:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574513.349590:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488781696:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574513.349604:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574513.349612:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574513/real 1788574513] req@ffff94d4049be680 x1875450488781696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574513.349620:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488781696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574513.349625:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574513.349627:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574513.349629:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488781696:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574513.349648:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488781824:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574513.349658:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574513.349665:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574513/real 1788574513] req@ffff94d4049bf800 x1875450488781824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574513.349672:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488781824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574513.349677:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574513.349679:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574513.349681:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488781824:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574513.349684:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488781952:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574513.349694:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574513.349699:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574513/real 1788574513] req@ffff94d4049bed80 x1875450488781952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574513.349707:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488781952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574513.349711:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574513.349713:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574513.349715:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488781952:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574513.349718:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488782080:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574513.349727:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574513.349732:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574513/real 1788574513] req@ffff94d4049bc700 x1875450488782080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574513.349740:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488782080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574513.349745:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574513.349747:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574513.349748:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488782080:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574513.861274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574514.309389:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574514.885278:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574515.333367:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574515.909383:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574516.357402:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574516.933329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574517.381347:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574517.957325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574518.405350:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574518.469315:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574518.469323:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574518.469327:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574518.469336:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5666 00000100:00080000:3.0:1788574518.469346:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574518.469352:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574518.469387:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488782208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574518.469411:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574518.469414:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574518.469416:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574518.469419:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5666 00000100:00080000:3.0:1788574518.469422:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574518.469424:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574518.469432:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574518.469442:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488782336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574518.469451:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574518.469454:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574518.469456:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574518.469461:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488782208:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574518.469462:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5666 00000100:00080000:3.0:1788574518.469468:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574518.469472:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574518.469475:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574518.469487:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488782464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574518.469494:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574518.469497:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574518.469498:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574518.469502:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5666 00000100:00080000:3.0:1788574518.469505:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574518.469508:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574518.469512:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574518.469520:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488782592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574518.469528:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574518.469530:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574518.469532:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788574518.469538:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574518.469538:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5666 00000100:00080000:3.0:1788574518.469545:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574518.469549:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574518.469552:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574518.469562:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488782720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574518.469580:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574518/real 1788574518] req@ffff94d408c56680 x1875450488782208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574518.469596:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488782208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574518.469605:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574518.469609:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574518.469612:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488782208:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574518.469650:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488782336:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574518.469671:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574518.469681:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574518/real 1788574518] req@ffff94d408c57b80 x1875450488782336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574518.469691:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488782336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574518.469704:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574518.469707:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574518.469709:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488782336:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574518.469714:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488782464:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574518.469727:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574518.469734:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574518/real 1788574518] req@ffff94d408c54a80 x1875450488782464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574518.469743:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488782464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574518.469760:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574518.469763:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574518.469765:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488782464:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574518.469770:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488782592:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574518.469785:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574518.469793:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574518/real 1788574518] req@ffff94d408c56d80 x1875450488782592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574518.469803:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488782592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574518.469809:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574518.469811:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574518.469814:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488782592:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574518.469820:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488782720:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574518.469841:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574518.469866:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574518/real 1788574518] req@ffff94d408c54e00 x1875450488782720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574518.469875:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488782720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574518.469881:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574518.469883:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574518.469886:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488782720:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574518.981284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574519.429343:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574520.005290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574520.453427:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574521.029311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574521.477412:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574522.053326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574522.501402:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574523.077320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574523.525384:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574523.589299:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574523.589308:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574523.589311:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574523.589320:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5671 00000100:00080000:3.0:1788574523.589329:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574523.589335:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574523.589366:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488782848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574523.589392:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574523.589395:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574523.589400:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574523.589403:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5671 00000100:00080000:3.0:1788574523.589406:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574523.589414:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574523.589423:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574523.589433:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488782976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574523.589441:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574523.589445:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788574523.589447:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488782848:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574523.589448:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574523.589455:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5671 00000100:00080000:3.0:1788574523.589458:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574523.589460:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574523.589463:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574523.589472:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488783104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574523.589484:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574523.589486:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574523.589488:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574523.589490:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5671 00000100:00080000:3.0:1788574523.589499:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574523.589502:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574523.589505:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574523.589509:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574523.589517:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488783232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574523.589524:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574523.589526:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574523.589528:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574523.589530:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5671 00000100:00080000:3.0:1788574523.589533:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574523.589536:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574523.589539:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574523.589545:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574523/real 1788574523] req@ffff94d408c55c00 x1875450488782848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574523.589554:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488783360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574523.589572:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488782848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574523.589582:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574523.589587:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574523.589591:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488782848:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574523.589620:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488782976:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574523.589645:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574523.589658:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574523/real 1788574523] req@ffff94d408c57100 x1875450488782976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574523.589671:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488782976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574523.589680:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574523.589684:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574523.589688:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488782976:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574523.589694:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488783104:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574523.589710:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574523.589720:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574523/real 1788574523] req@ffff94d408c56a00 x1875450488783104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574523.589732:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488783104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574523.589741:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574523.589744:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574523.589748:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488783104:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574523.589754:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488783232:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574523.589775:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574523.589784:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574523/real 1788574523] req@ffff94d408c57800 x1875450488783232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574523.589792:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488783232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574523.589797:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574523.589799:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574523.589802:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488783232:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574523.589805:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488783360:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574523.589817:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574523.589823:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574523/real 1788574523] req@ffff94d408c54380 x1875450488783360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574523.589830:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488783360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574523.589835:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574523.589837:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574523.589839:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488783360:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574524.101354:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574524.549495:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574525.125361:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574525.573365:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574526.149301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574526.597382:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574527.173305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574527.621357:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574528.197320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574528.645407:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574528.709379:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574528.709393:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574528.709399:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574528.709411:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5677 00000100:00080000:1.0:1788574528.709425:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574528.709441:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574528.709490:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488783488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574528.709521:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574528.709526:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574528.709528:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574528.709532:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5677 00000100:00080000:1.0:1788574528.709536:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574528.709540:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574528.709551:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574528.709565:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488783616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574528.709572:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488783488:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788574528.709582:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574528.709587:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574528.709589:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574528.709594:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5677 00000100:00080000:1.0:1788574528.709598:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574528.709602:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574528.709607:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574528.709620:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488783744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574528.709636:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574528.709641:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574528.709643:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574528.709647:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5677 00000100:00080000:1.0:1788574528.709651:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788574528.709655:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788574528.709658:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574528.709664:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574528.709676:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488783872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574528.709687:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574528.709690:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574528.709692:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574528.709696:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5677 00000100:00080000:1.0:1788574528.709699:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788574528.709701:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574528/real 1788574528] req@ffff94d4049bc000 x1875450488783488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574528.709706:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574528.709713:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574528.709724:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488783488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574528.709729:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488784000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574528.709738:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574528.709745:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574528.709751:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488783488:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574528.709776:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488783616:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574528.709792:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574528.709800:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574528/real 1788574528] req@ffff94d4049bfb80 x1875450488783616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574528.709808:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488783616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574528.709814:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574528.709817:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574528.709820:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488783616:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574528.709823:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488783744:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574528.709833:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574528.709839:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574528/real 1788574528] req@ffff94d4049bea00 x1875450488783744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574528.709846:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488783744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574528.709852:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574528.709854:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574528.709856:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488783744:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574528.709859:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488783872:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574528.709869:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574528.709874:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574528/real 1788574528] req@ffff94d4049bf480 x1875450488783872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574528.709883:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488783872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574528.709888:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574528.709889:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574528.709891:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488783872:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574528.709894:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488784000:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574528.709904:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574528.709909:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574528/real 1788574528] req@ffff94d4049bc380 x1875450488784000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574528.709916:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc380 x1875450488784000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574528.709922:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574528.709924:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574528.709925:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488784000:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574529.221314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574529.669365:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574530.245296:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574530.693351:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574531.269289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574531.717358:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574532.293297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574532.741357:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574533.317284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574533.765370:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574533.829427:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574533.829439:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574533.829444:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574533.829459:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5682 00000100:00080000:3.0:1788574533.829474:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574533.829483:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574533.829525:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488784128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574533.829552:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574533.829556:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574533.829557:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574533.829561:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5682 00000100:00080000:3.0:1788574533.829563:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574533.829566:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574533.829575:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574533.829588:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488784256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574533.829595:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488784128:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574533.829599:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574533.829612:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574533.829613:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574533.829618:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5682 00000100:00080000:3.0:1788574533.829622:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574533.829625:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574533.829630:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574533.829641:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488784384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574533.829649:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574533.829652:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574533.829654:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574533.829657:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5682 00000100:00080000:3.0:1788574533.829660:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574533.829663:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574533.829667:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574533.829675:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488784512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574533.829681:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574533.829688:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574533.829691:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574533.829693:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574533.829697:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5682 00000100:00080000:3.0:1788574533.829705:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574533.829719:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574533/real 1788574533] req@ffff94d408c55880 x1875450488784128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574533.829736:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488784128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574533.829754:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574533.829758:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574533.829762:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488784128:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574533.829771:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574533.829777:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574533.829787:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488784640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574533.829791:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488784256:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574533.829819:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574533.829832:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574533/real 1788574533] req@ffff94d408c55180 x1875450488784256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574533.829841:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488784256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574533.829847:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574533.829851:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574533.829853:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488784256:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574533.829857:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488784384:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574533.829873:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574533.829885:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574533/real 1788574533] req@ffff94d408c54000 x1875450488784384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574533.829899:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488784384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574533.829907:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574533.829909:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574533.829911:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488784384:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574533.829916:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488784512:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574533.829931:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574533.829943:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574533/real 1788574533] req@ffff94d408c56300 x1875450488784512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574533.829957:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488784512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574533.829967:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574533.829970:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574533.829975:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488784512:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574533.829997:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488784640:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574533.830032:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574533.830046:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574533/real 1788574533] req@ffff94d408c55f80 x1875450488784640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574533.830062:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488784640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574533.830087:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574533.830091:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574533.830094:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488784640:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574534.341319:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574534.789387:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574535.365382:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574535.813450:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574536.389333:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574536.837384:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574537.413282:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574537.861448:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574538.437280:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574538.885360:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574538.949331:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574538.949339:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574538.949343:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574538.949351:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5687 00000100:00080000:1.0:1788574538.949360:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574538.949367:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574538.949400:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488784768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574538.949425:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574538.949428:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574538.949430:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574538.949434:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5687 00000100:00080000:1.0:1788574538.949438:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574538.949443:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574538.949453:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574538.949467:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488784896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574538.949489:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574538.949492:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488784768:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788574538.949494:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574538.949496:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574538.949507:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5687 00000100:00080000:1.0:1788574538.949513:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574538.949516:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574538.949520:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574538.949534:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488785024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574538.949540:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574538.949542:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574538.949543:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574538.949546:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5687 00000100:00080000:1.0:1788574538.949547:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574538.949550:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574538.949553:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574538.949559:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488785152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574538.949568:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574538.949570:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574538.949571:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574538.949573:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5687 00000100:00080000:1.0:1788574538.949577:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574538.949580:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574538.949583:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574538.949589:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488785280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574538.949606:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574538.949660:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574538/real 1788574538] req@ffff94d4049bd180 x1875450488784768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574538.949682:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488784768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574538.949702:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574538.949707:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574538.949712:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488784768:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574538.949743:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488784896:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574538.949781:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574538.949796:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574538/real 1788574538] req@ffff94d4049bce00 x1875450488784896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574538.949810:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488784896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574538.949819:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574538.949823:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574538.949827:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488784896:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574538.949834:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488785024:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574538.949855:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574538.949866:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574538/real 1788574538] req@ffff94d4049bc380 x1875450488785024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574538.949879:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc380 x1875450488785024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574538.949888:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574538.949892:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574538.949896:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488785024:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574538.949903:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488785152:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574538.949924:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574538.949935:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574538/real 1788574538] req@ffff94d4049bf480 x1875450488785152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574538.949947:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488785152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574538.949956:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574538.949960:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574538.949963:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488785152:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574538.949970:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488785280:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574538.949989:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574538.949999:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574538/real 1788574538] req@ffff94d4049bea00 x1875450488785280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574538.950013:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488785280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574538.950022:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574538.950025:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574538.950068:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488785280:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574539.461345:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574539.909404:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574540.485312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574540.933440:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574541.509302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574541.957377:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574542.533288:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574542.981404:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574543.557367:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574544.005392:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574544.069315:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574544.069323:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574544.069327:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574544.069336:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5692 00000100:00080000:0.0:1788574544.069345:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574544.069352:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574544.069384:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488785408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574544.069405:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574544.069408:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574544.069409:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574544.069412:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5692 00000100:00080000:0.0:1788574544.069415:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574544.069417:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574544.069425:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574544.069434:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488785536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574544.069441:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574544.069444:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574544.069445:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574544.069448:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5692 00000100:00080000:0.0:1788574544.069449:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574544.069452:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574544.069455:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574544.069466:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488785408:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574544.069471:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488785664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574544.069480:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574544.069482:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574544.069484:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574544.069486:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5692 00000100:00080000:0.0:1788574544.069488:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574544.069491:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574544.069494:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574544.069501:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488785792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574544.069513:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574544.069515:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574544.069516:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574544.069518:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5692 00000100:00080000:0.0:1788574544.069521:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574544.069523:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574544.069526:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574544.069533:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488785920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574544.069541:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574544.069583:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574544/real 1788574544] req@ffff94d408d3e680 x1875450488785408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574544.069604:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488785408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574544.069617:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574544.069624:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574544.069629:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488785408:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574544.069640:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488785536:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574544.069667:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574544.069685:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574544/real 1788574544] req@ffff94d408d3ea00 x1875450488785536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574544.069699:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488785536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574544.069708:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574544.069711:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574544.069715:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488785536:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574544.069743:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488785664:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574544.069765:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574544.069776:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574544/real 1788574544] req@ffff94d408d3df80 x1875450488785664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574544.069788:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488785664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574544.069816:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574544.069821:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574544.069825:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488785664:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574544.069831:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488785792:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574544.069853:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574544.069866:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574544/real 1788574544] req@ffff94d408d3d180 x1875450488785792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574544.069879:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488785792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574544.069888:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574544.069891:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574544.069904:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488785792:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574544.069912:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488785920:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574544.069934:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574544.069947:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574544/real 1788574544] req@ffff94d408d3c000 x1875450488785920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574544.069977:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488785920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574544.069987:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574544.069990:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574544.069994:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488785920:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574544.581338:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574545.029377:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574545.605302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574546.053394:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574546.629266:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574547.077364:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574547.653284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574548.101329:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574548.677284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574549.125328:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574549.189293:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574549.189301:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574549.189305:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574549.189313:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5697 00000100:00080000:3.0:1788574549.189322:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574549.189328:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574549.189371:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488786048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574549.189394:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574549.189396:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574549.189398:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574549.189400:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5697 00000100:00080000:3.0:1788574549.189402:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574549.189405:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574549.189412:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574549.189420:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488786176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574549.189429:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574549.189432:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574549.189433:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574549.189438:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5697 00000100:00100000:2.0:1788574549.189439:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488786048:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574549.189440:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574549.189447:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574549.189454:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574549.189469:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488786304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574549.189480:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574549.189484:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574549.189486:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574549.189491:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5697 00000100:00080000:3.0:1788574549.189503:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574549.189508:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574549.189512:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574549.189531:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488786432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574549.189542:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574549.189546:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1788574549.189548:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488786176:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788574549.189549:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574549.189559:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5697 00000100:00080000:3.0:1788574549.189564:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574549.189569:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574549.189575:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574549.189594:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488786304:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574549.189601:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488786560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574549.189628:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488786432:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574549.189657:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488786560:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574549.701314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574550.149339:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574550.725282:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574551.173356:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574551.749270:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574552.197387:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788574552.261563:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:1.0:1788574552.261628:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788574552.261651:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788574552.261711:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788574552.261723:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788574552.261734:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788574552.261743:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:2.0:1788574552.261853:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574549/real 1788574552] req@ffff94d408c54000 x1875450488786560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574552.261903:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488786560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574552.261921:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574552.261927:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574552.261933:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488786560:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574552.261944:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574549/real 1788574552] req@ffff94d408c57480 x1875450488786176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574552.261962:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488786176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574552.261974:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574552.261978:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574552.261983:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488786176:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574552.261993:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574549/real 1788574552] req@ffff94d408c55f80 x1875450488786304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574552.262008:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488786304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574552.262018:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574552.262022:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574552.262032:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488786304:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574552.262042:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574549/real 1788574552] req@ffff94d408c56300 x1875450488786432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574552.262057:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488786432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574552.262065:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574552.262069:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574552.262073:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488786432:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574552.262083:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574549/real 1788574552] req@ffff94d408c54700 x1875450488786048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574552.262098:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488786048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574552.262123:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574552.262142:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574552.262146:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488786048:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574552.773259:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574553.221335:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574553.797277:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574554.245364:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574554.309421:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574554.309433:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574554.309437:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574554.309449:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5702 00000100:00080000:1.0:1788574554.309463:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574554.309473:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574554.309523:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488786688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574554.309559:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574554.309562:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574554.309563:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574554.309567:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5702 00000100:00080000:1.0:1788574554.309570:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574554.309574:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574554.309584:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574554.309598:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488786816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574554.309605:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574554.309608:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574554.309610:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574554.309614:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5702 00000100:00080000:1.0:1788574554.309617:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574554.309628:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574554.309628:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488786688:192.168.202.125@tcp:250:kworker.0 00000008:00100000:1.0:1788574554.309638:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574554.309653:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488786944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574554.309664:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574554.309668:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574554.309671:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574554.309676:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5702 00000100:00080000:1.0:1788574554.309680:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574554.309685:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574554.309698:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574554.309711:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488787072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574554.309722:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574554.309726:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574554.309728:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574554.309732:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5702 00000100:00080000:1.0:1788574554.309736:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574554.309744:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:2.0:1788574554.309747:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:1.0:1788574554.309753:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574554.309767:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488787200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574554.309806:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574554/real 1788574554] req@ffff94d4049bfb80 x1875450488786688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574554.309834:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488786688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574554.309849:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574554.309854:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574554.309860:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488786688:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574554.309869:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488786816:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574554.309896:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574554.309926:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574554/real 1788574554] req@ffff94d4049bc000 x1875450488786816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574554.309941:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488786816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574554.309951:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574554.309954:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574554.309958:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488786816:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574554.309993:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488786944:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574554.310018:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574554.310031:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574554/real 1788574554] req@ffff94d4049bc700 x1875450488786944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574554.310044:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488786944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574554.310054:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574554.310058:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574554.310061:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488786944:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574554.310069:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488787072:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574554.310099:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574554.310113:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574554/real 1788574554] req@ffff94d4049bed80 x1875450488787072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574554.310145:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488787072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574554.310155:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574554.310158:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574554.310162:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488787072:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574554.310169:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488787200:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574554.310191:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574554.310249:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574554/real 1788574554] req@ffff94d4049bf800 x1875450488787200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574554.310263:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488787200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574554.310277:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574554.310281:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574554.310285:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488787200:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574554.821283:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574555.269450:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574555.845309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574556.293387:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574556.869286:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574557.317372:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574557.893368:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574558.341336:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574558.917479:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574559.365395:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574559.429296:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574559.429304:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574559.429308:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574559.429316:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5707 00000100:00080000:1.0:1788574559.429326:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574559.429334:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574559.429369:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488787328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574559.429389:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574559.429392:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574559.429393:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574559.429396:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5707 00000100:00080000:1.0:1788574559.429398:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574559.429401:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574559.429408:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574559.429417:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488787456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574559.429426:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574559.429428:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574559.429431:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574559.429433:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488787328:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788574559.429435:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5707 00000100:00080000:1.0:1788574559.429441:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574559.429445:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574559.429448:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574559.429457:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488787584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574559.429468:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574559.429471:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574559.429474:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574559.429478:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5707 00000100:00080000:1.0:1788574559.429482:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574559.429488:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574559.429494:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574559.429497:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574559.429513:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488787712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574559.429526:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574559.429531:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574559/real 1788574559] req@ffff94d4049be680 x1875450488787328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574559.429532:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574559.429552:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574559.429561:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5707 00000100:00080000:1.0:1788574559.429568:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574559.429575:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574559.429580:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574559.429582:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488787328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574559.429594:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488787840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574559.429595:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574559.429606:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574559.429615:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488787328:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574559.429640:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488787456:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574559.429686:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574559.429699:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574559/real 1788574559] req@ffff94d4049bf100 x1875450488787456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574559.429713:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488787456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574559.429723:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574559.429726:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574559.429730:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488787456:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574559.429737:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488787584:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574559.429754:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574559.429762:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574559/real 1788574559] req@ffff94d4049bca80 x1875450488787584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574559.429770:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488787584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574559.429776:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574559.429778:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574559.429781:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488787584:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574559.429784:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488787712:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574559.429809:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574559.429816:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574559/real 1788574559] req@ffff94d4049bd500 x1875450488787712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574559.429824:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488787712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574559.429830:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574559.429832:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574559.429834:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488787712:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574559.429838:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488787840:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574559.429848:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574559.429854:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574559/real 1788574559] req@ffff94d4049bdc00 x1875450488787840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574559.429861:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488787840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574559.429866:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574559.429868:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574559.429870:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488787840:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574559.941559:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574560.389330:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574560.965793:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574561.413330:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574561.989527:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574562.437381:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574563.013446:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574563.461334:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574564.037308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574564.485361:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574564.549348:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574564.549356:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574564.549359:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574564.549368:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5712 00000100:00080000:0.0:1788574564.549377:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574564.549383:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574564.549416:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488787968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574564.549445:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574564.549448:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574564.549453:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574564.549456:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5712 00000100:00080000:0.0:1788574564.549458:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574564.549461:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574564.549469:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574564.549479:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488788096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574564.549488:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574564.549489:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488787968:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788574564.549492:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574564.549497:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574564.549501:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5712 00000100:00080000:0.0:1788574564.549503:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574564.549505:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574564.549508:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574564.549516:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488788224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574564.549523:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574564.549526:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574564.549527:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574564.549530:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5712 00000100:00080000:0.0:1788574564.549532:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574564.549535:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574564.549539:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574564.549546:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488788352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574564.549550:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788574564.549562:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574564.549566:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574564.549569:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574564.549575:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5712 00000100:00080000:0.0:1788574564.549579:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574564.549583:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574564.549589:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574564.549605:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488788480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574564.549606:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574564/real 1788574564] req@ffff94d408d3c380 x1875450488787968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574564.549637:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488787968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574564.549646:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574564.549649:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574564.549653:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488787968:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574564.549677:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488788096:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574564.549703:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574564.549715:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574564/real 1788574564] req@ffff94d408d3f480 x1875450488788096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574564.549724:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488788096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574564.549735:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574564.549737:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574564.549739:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488788096:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574564.549743:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488788224:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574564.549756:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574564.549762:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574564/real 1788574564] req@ffff94d408d3c700 x1875450488788224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574564.549770:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488788224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574564.549775:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574564.549777:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574564.549779:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488788224:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574564.549782:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488788352:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574564.549793:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574564.549799:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574564/real 1788574564] req@ffff94d408d3e300 x1875450488788352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574564.549811:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488788352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574564.549819:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574564.549823:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574564.549827:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488788352:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574564.549833:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488788480:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574564.549855:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574564.549864:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574564/real 1788574564] req@ffff94d408d3dc00 x1875450488788480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574564.549872:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488788480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574564.549877:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574564.549879:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574564.549881:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488788480:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574565.061355:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574565.509411:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574566.085352:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574566.533381:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574567.109258:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574567.557368:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574568.133268:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574568.581368:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574569.157272:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574569.605372:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574569.669322:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574569.669332:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574569.669337:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574569.669348:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5717 00000100:00080000:0.0:1788574569.669358:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574569.669368:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574569.669405:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ed80 x1875450488788608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574569.669442:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574569.669454:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574569.669456:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574569.669461:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5717 00000100:00080000:0.0:1788574569.669466:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574569.669471:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574569.669481:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574569.669497:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488788608:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574569.669498:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488788736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574569.669526:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574569.669529:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574569.669531:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574569.669534:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5717 00000100:00080000:0.0:1788574569.669539:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574569.669543:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574569.669548:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574569.669564:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488788864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574569.669564:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788574569.669575:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574569.669578:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574569.669579:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574569.669583:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5717 00000100:00080000:0.0:1788574569.669587:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574569.669590:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574569.669595:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574569.669604:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488788992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574569.669611:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574569.669613:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574569.669614:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574569.669616:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5717 00000100:00080000:0.0:1788574569.669620:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574569.669623:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574569.669633:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574569.669642:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488789120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574569.669791:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574569/real 1788574569] req@ffff94d408d3ed80 x1875450488788608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574569.669808:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ed80 x1875450488788608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574569.669817:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574569.669820:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574569.669864:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488788608:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574569.669888:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488788736:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574569.669919:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574569.669930:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574569/real 1788574569] req@ffff94d408d3d880 x1875450488788736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574569.669939:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488788736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574569.669944:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574569.669947:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574569.669950:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488788736:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574569.669954:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488788864:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574569.669968:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574569.669979:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574569/real 1788574569] req@ffff94d408d3dc00 x1875450488788864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574569.669991:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488788864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574569.670008:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574569.670012:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574569.670016:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488788864:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574569.670024:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488788992:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574569.670044:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574569.670086:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574569/real 1788574569] req@ffff94d408d3e300 x1875450488788992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574569.670100:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488788992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574569.670109:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574569.670113:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574569.670117:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488788992:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574569.670131:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488789120:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574569.670152:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574569.670164:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574569/real 1788574569] req@ffff94d408d3c700 x1875450488789120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574569.670178:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488789120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574569.670187:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574569.670191:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574569.670195:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488789120:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574570.181331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574570.629411:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574571.205271:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574571.653351:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574572.229253:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574572.677380:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574573.253377:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574573.701356:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574574.277355:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574574.725350:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574574.789347:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574574.789355:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574574.789359:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574574.789368:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5723 00000100:00080000:1.0:1788574574.789377:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574574.789383:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574574.789415:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd880 x1875450488789248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574574.789438:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574574.789441:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574574.789442:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574574.789445:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5723 00000100:00080000:1.0:1788574574.789447:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574574.789450:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574574.789458:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574574.789467:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488789376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574574.789474:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574574.789477:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574574.789478:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574574.789481:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5723 00000100:00080000:1.0:1788574574.789482:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574574.789486:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574574.789489:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574574.789501:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488789504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574574.789509:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488789248:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788574574.789512:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574574.789520:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574574.789525:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574574.789531:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5723 00000100:00080000:1.0:1788574574.789533:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574574.789535:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574574.789538:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574574.789547:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488789632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574574.789554:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574574.789556:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574574.789557:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574574.789559:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5723 00000100:00080000:1.0:1788574574.789562:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574574.789564:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574574.789568:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574574.789577:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488789760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574574.789611:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574574.789665:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574574/real 1788574574] req@ffff94d4049bd880 x1875450488789248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574574.789697:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd880 x1875450488789248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574574.789707:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574574.789710:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574574.789714:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488789248:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574574.789721:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488789376:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574574.789750:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574574.789761:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574574/real 1788574574] req@ffff94d4049be300 x1875450488789376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574574.789775:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488789376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574574.789784:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574574.789789:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574574.789792:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488789376:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574574.789823:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488789504:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574574.789841:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574574.789859:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574574/real 1788574574] req@ffff94d4049bdc00 x1875450488789504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574574.789868:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488789504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574574.789879:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574574.789882:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574574.789884:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488789504:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574574.789889:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488789632:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574574.789902:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574574.789909:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574574/real 1788574574] req@ffff94d4049bd500 x1875450488789632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574574.789917:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488789632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574574.789923:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574574.789925:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574574.789927:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488789632:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574574.789930:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488789760:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574574.789947:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574574.789954:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574574/real 1788574574] req@ffff94d4049bca80 x1875450488789760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574574.789961:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488789760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574574.789967:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574574.789969:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574574.789971:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488789760:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574575.301243:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574575.749367:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574576.325327:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574576.773390:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574577.349362:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574577.797371:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574578.373325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574578.821377:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574579.397273:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574579.845359:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574579.909339:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574579.909348:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574579.909351:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574579.909360:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5728 00000100:00080000:3.0:1788574579.909369:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574579.909375:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574579.909406:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488789888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574579.909429:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574579.909432:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574579.909434:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574579.909436:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5728 00000100:00080000:3.0:1788574579.909439:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574579.909441:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574579.909448:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574579.909456:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488790016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574579.909468:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574579.909470:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574579.909472:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574579.909474:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5728 00000100:00080000:3.0:1788574579.909476:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574579.909478:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574579.909481:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574579.909488:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488789888:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574579.909489:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488790144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574579.909504:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574579.909506:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574579.909507:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574579.909509:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5728 00000100:00080000:3.0:1788574579.909511:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574579.909513:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574579.909516:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574579.909532:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488790272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574579.909539:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574579.909541:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574579.909542:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574579.909544:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5728 00000100:00080000:3.0:1788574579.909553:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574579.909555:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574579.909559:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574579.909564:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574579.909568:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488790400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574579.909615:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574579/real 1788574579] req@ffff94d408c55180 x1875450488789888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574579.909634:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488789888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574579.909645:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574579.909650:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574579.909654:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488789888:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574579.909662:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488790016:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574579.909688:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574579.909704:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574579/real 1788574579] req@ffff94d408c55880 x1875450488790016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574579.909718:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488790016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574579.909728:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574579.909731:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574579.909735:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488790016:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574579.909759:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488790144:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574579.909782:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574579.909794:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574579/real 1788574579] req@ffff94d408c54380 x1875450488790144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574579.909806:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488790144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574579.909814:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574579.909817:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574579.909820:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488790144:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574579.909826:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488790272:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574579.909954:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574579.909968:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574579/real 1788574579] req@ffff94d408c57800 x1875450488790272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574579.909981:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488790272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574579.910023:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574579.910028:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574579.910031:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488790272:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574579.910038:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488790400:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574579.910058:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574579.910069:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574579/real 1788574579] req@ffff94d408c56a00 x1875450488790400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574579.910163:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488790400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574579.910171:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574579.910174:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574579.910178:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488790400:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574580.421282:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574580.869360:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574581.445289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574581.893384:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574582.469336:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574582.917388:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574583.493282:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574583.941367:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574584.517297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574584.965373:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574585.029305:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574585.029316:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574585.029321:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574585.029331:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5733 00000100:00080000:0.0:1788574585.029342:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574585.029349:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574585.029383:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488790528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574585.029412:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574585.029416:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574585.029418:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574585.029422:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5733 00000100:00080000:0.0:1788574585.029425:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574585.029429:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574585.029437:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574585.029450:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488790656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574585.029462:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574585.029466:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488790528:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788574585.029472:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574585.029475:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574585.029480:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5733 00000100:00080000:0.0:1788574585.029484:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574585.029488:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574585.029493:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574585.029505:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488790784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574585.029515:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574585.029518:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574585.029520:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788574585.029526:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788574585.029530:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5733 00000100:00080000:0.0:1788574585.029535:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574585.029540:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574585.029546:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574585.029557:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488790912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574585.029567:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574585.029570:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574585/real 1788574585] req@ffff94d408d3f480 x1875450488790528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574585.029571:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574585.029575:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574585.029582:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5733 00000100:00080000:0.0:1788574585.029589:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574585.029596:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574585.029602:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574585.029608:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488790528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574585.029615:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488791040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574585.029630:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574585.029635:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574585.029640:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488790528:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574585.029661:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488790656:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574585.029680:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574585.029689:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574585/real 1788574585] req@ffff94d408d3c380 x1875450488790656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574585.029709:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488790656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574585.029715:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574585.029717:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574585.029720:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488790656:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574585.029724:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488790784:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574585.029736:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574585.029743:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574585/real 1788574585] req@ffff94d408d3c000 x1875450488790784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574585.029750:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488790784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574585.029756:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574585.029758:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574585.029760:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488790784:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574585.029763:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488790912:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574585.029774:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574585.029779:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574585/real 1788574585] req@ffff94d408d3d180 x1875450488790912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574585.029787:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488790912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574585.029792:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574585.029793:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574585.029795:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488790912:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574585.029798:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488791040:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574585.029808:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574585.029813:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574585/real 1788574585] req@ffff94d408d3df80 x1875450488791040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574585.029821:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488791040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574585.029825:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574585.029827:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574585.029829:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488791040:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574585.541354:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574585.989382:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574586.565293:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574587.013341:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574587.589309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574588.037345:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574588.613264:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574589.061379:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574589.637315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574590.085359:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574590.149360:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574590.149370:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574590.149375:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574590.149386:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5738 00000100:00080000:1.0:1788574590.149398:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574590.149405:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574590.149450:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488791168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574590.149484:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574590.149489:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574590.149491:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574590.149496:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5738 00000100:00080000:1.0:1788574590.149500:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574590.149504:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574590.149516:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574590.149535:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488791296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574590.149545:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488791168:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788574590.149558:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574590.149563:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574590.149567:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574590.149574:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5738 00000100:00080000:1.0:1788574590.149577:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574590.149581:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574590.149586:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574590.149598:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488791424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574590.149605:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574590.149608:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574590.149609:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574590.149612:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5738 00000100:00080000:1.0:1788574590.149614:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574590.149620:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574590.149627:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574590.149627:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574590.149641:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488791552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574590.149648:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574590.149651:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574590.149653:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574590.149656:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5738 00000100:00080000:1.0:1788574590.149660:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574590.149662:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574590.149667:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574590.149678:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488791680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574590.149681:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574590/real 1788574590] req@ffff94d4049bf100 x1875450488791168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574590.149736:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488791168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574590.149750:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574590.149754:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574590.149758:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488791168:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574590.149779:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488791296:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574590.149800:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574590.149808:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574590/real 1788574590] req@ffff94d4049be680 x1875450488791296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574590.149817:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488791296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574590.149823:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574590.149825:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574590.149827:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488791296:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574590.149831:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488791424:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574590.149842:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574590.149848:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574590/real 1788574590] req@ffff94d4049bf800 x1875450488791424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574590.149856:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488791424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574590.149861:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574590.149863:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574590.149865:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488791424:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574590.149869:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488791552:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574590.149880:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574590.149886:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574590/real 1788574590] req@ffff94d4049bed80 x1875450488791552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574590.149893:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488791552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574590.149898:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574590.149900:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574590.149902:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488791552:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574590.149905:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488791680:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574590.149914:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574590.149920:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574590/real 1788574590] req@ffff94d4049bc700 x1875450488791680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574590.149944:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488791680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574590.149949:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574590.149952:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574590.149954:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488791680:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574590.661361:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574591.109356:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574591.685265:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574592.133355:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574592.709292:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574593.157348:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574593.733270:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574594.181397:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574594.757305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574595.205326:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574595.269325:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574595.269333:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574595.269336:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574595.269345:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5743 00000100:00080000:1.0:1788574595.269358:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574595.269364:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574595.269396:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488791808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574595.269419:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574595.269422:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574595.269423:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574595.269426:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5743 00000100:00080000:1.0:1788574595.269428:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574595.269431:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574595.269438:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574595.269446:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488791936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574595.269455:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574595.269457:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574595.269460:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574595.269463:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488791808:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788574595.269465:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5743 00000100:00080000:1.0:1788574595.269472:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574595.269475:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574595.269479:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574595.269488:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488792064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574595.269494:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574595.269496:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574595.269498:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574595.269500:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5743 00000100:00080000:1.0:1788574595.269503:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574595.269505:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574595.269508:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574595.269515:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488792192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574595.269522:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574595.269525:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574595.269526:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574595.269528:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5743 00000100:00080000:1.0:1788574595.269531:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788574595.269535:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788574595.269538:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574595.269542:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574595.269551:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488792320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574595.269573:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574595/real 1788574595] req@ffff94d4049bc000 x1875450488791808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574595.269589:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488791808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574595.269598:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574595.269601:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574595.269604:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488791808:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574595.269622:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488791936:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574595.269639:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574595.269647:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574595/real 1788574595] req@ffff94d4049bfb80 x1875450488791936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574595.269655:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488791936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574595.269661:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574595.269663:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574595.269665:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488791936:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574595.269668:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488792064:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574595.269678:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574595.269684:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574595/real 1788574595] req@ffff94d4049bea00 x1875450488792064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574595.269691:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488792064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574595.269701:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574595.269703:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574595.269705:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488792064:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574595.269708:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488792192:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574595.269718:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574595.269723:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574595/real 1788574595] req@ffff94d4049bf480 x1875450488792192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574595.269730:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488792192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574595.269738:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574595.269742:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574595.269745:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488792192:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574595.269751:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488792320:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574595.269770:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574595.269782:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574595/real 1788574595] req@ffff94d4049bc380 x1875450488792320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574595.269794:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc380 x1875450488792320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574595.269799:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574595.269801:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574595.269804:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488792320:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574595.781262:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574596.229367:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574596.805319:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574597.253370:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574597.829305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574598.277437:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574598.853329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574599.301392:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574599.877378:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574600.325445:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574600.389410:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574600.389421:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574600.389426:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574600.389443:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5748 00000100:00080000:3.0:1788574600.389474:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574600.389484:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574600.389537:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488792448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574600.389563:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574600.389566:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574600.389568:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574600.389571:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5748 00000100:00080000:3.0:1788574600.389574:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574600.389578:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574600.389588:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574600.389600:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488792576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574600.389609:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574600.389611:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574600.389612:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574600.389617:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5748 00000100:00100000:2.0:1788574600.389619:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488792448:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574600.389620:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574600.389624:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574600.389631:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574600.389638:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488792704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574600.389644:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574600.389646:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574600.389648:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574600.389650:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5748 00000100:00080000:3.0:1788574600.389652:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574600.389655:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574600.389658:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574600.389666:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488792832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574600.389672:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574600.389674:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574600.389675:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574600.389679:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5748 00000100:00080000:3.0:1788574600.389700:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574600.389703:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574600.389707:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574600.389712:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574600.389717:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488792960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574600.389754:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574600/real 1788574600] req@ffff94d408c57100 x1875450488792448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574600.389770:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488792448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574600.389780:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574600.389783:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574600.389793:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488792448:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574600.389819:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488792576:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574600.389836:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574600.389845:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574600/real 1788574600] req@ffff94d408c55c00 x1875450488792576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574600.389854:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488792576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574600.389860:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574600.389876:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574600.389879:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488792576:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574600.389883:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488792704:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574600.389899:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574600.389907:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574600/real 1788574600] req@ffff94d408c54e00 x1875450488792704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574600.389924:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488792704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574600.389931:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574600.389933:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574600.389935:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488792704:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574600.389940:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488792832:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574600.389954:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574600.389960:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574600/real 1788574600] req@ffff94d408c56d80 x1875450488792832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574600.389968:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488792832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574600.389973:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574600.389975:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574600.389977:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488792832:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574600.389981:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488792960:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574600.389991:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574600.389997:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574600/real 1788574600] req@ffff94d408c54a80 x1875450488792960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574600.390004:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488792960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574600.390010:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574600.390013:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574600.390024:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488792960:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574600.901287:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574601.349432:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574601.925347:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574602.373403:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574602.949317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574603.397397:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574603.973311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574604.421345:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574604.997334:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574605.445375:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574605.509360:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574605.509370:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574605.509375:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574605.509387:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5753 00000100:00080000:3.0:1788574605.509399:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574605.509407:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574605.509451:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488793088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574605.509486:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574605.509491:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574605.509494:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574605.509498:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5753 00000100:00080000:3.0:1788574605.509503:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574605.509508:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574605.509519:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574605.509532:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488793216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574605.509542:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574605.509546:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574605.509547:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574605.509552:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5753 00000100:00100000:2.0:1788574605.509562:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488793088:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574605.509569:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574605.509578:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574605.509584:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574605.509605:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488793344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574605.509617:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574605.509620:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574605.509621:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574605.509625:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5753 00000100:00080000:3.0:1788574605.509630:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574605.509634:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:2.0:1788574605.509639:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:3.0:1788574605.509641:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574605.509655:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488793472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574605.509665:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574605.509668:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574605.509670:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574605.509676:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5753 00000100:00080000:3.0:1788574605.509679:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574605.509688:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574605.509689:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574605/real 1788574605] req@ffff94d408c57b80 x1875450488793088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788574605.509699:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574605.509720:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488793600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574605.509726:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488793088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574605.509769:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574605.509776:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574605.509781:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488793088:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574605.509808:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488793216:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574605.509852:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574605.509864:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574605/real 1788574605] req@ffff94d408c56680 x1875450488793216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574605.509881:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488793216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574605.509888:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574605.509891:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574605.509896:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488793216:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574605.509904:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488793344:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574605.509921:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574605.509929:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574605/real 1788574605] req@ffff94d408c54a80 x1875450488793344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574605.509942:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488793344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574605.509949:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574605.509953:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574605.509955:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488793344:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574605.509960:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488793472:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574605.509972:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574605.509979:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574605/real 1788574605] req@ffff94d408c56d80 x1875450488793472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574605.509987:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488793472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574605.509993:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574605.509995:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574605.509997:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488793472:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574605.510001:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488793600:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574605.510018:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574605.510028:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574605/real 1788574605] req@ffff94d408c54e00 x1875450488793600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574605.510036:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488793600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574605.510042:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574605.510047:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574605.510052:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488793600:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574606.021324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574606.469347:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574607.045307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574607.493356:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574608.069337:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574608.517406:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574609.093378:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574609.541348:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574610.117311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574610.565389:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574610.629267:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574610.629278:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574610.629282:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574610.629293:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5758 00000100:00080000:3.0:1788574610.629304:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574610.629312:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574610.629350:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488793728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574610.629377:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574610.629381:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574610.629383:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574610.629387:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5758 00000100:00080000:3.0:1788574610.629391:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574610.629396:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574610.629405:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574610.629418:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488793728:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574610.629421:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488793856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574610.629434:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574610.629439:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574610.629441:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574610.629445:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5758 00000100:00080000:3.0:1788574610.629448:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574610.629452:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574610.629456:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574610.629468:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488793984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574610.629477:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574610.629480:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574610.629482:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574610.629486:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5758 00000100:00080000:3.0:1788574610.629490:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574610.629493:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:2.0:1788574610.629500:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:3.0:1788574610.629500:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574610.629515:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488794112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574610.629525:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574610.629529:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574610.629530:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574610.629534:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5758 00000100:00080000:3.0:1788574610.629539:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574610.629543:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574610.629550:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574610/real 1788574610] req@ffff94d408c55c00 x1875450488793728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788574610.629550:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574610.629574:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488794240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574610.629578:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488793728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574610.629604:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574610.629609:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574610.629614:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488793728:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574610.629639:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488793856:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574610.629665:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574610.629678:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574610/real 1788574610] req@ffff94d408c57100 x1875450488793856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574610.629692:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488793856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574610.629708:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574610.629712:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574610.629716:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488793856:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574610.629722:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488793984:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574610.629744:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574610.629756:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574610/real 1788574610] req@ffff94d408c56a00 x1875450488793984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574610.629770:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488793984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574610.629778:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574610.629782:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574610.629786:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488793984:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574610.629792:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488794112:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574610.629847:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574610.629859:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574610/real 1788574610] req@ffff94d408c57800 x1875450488794112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574610.629873:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488794112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574610.629882:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574610.629886:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574610.629890:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488794112:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574610.629897:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488794240:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574610.629919:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574610.629931:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574610/real 1788574610] req@ffff94d408c54380 x1875450488794240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574610.629945:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488794240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574610.629954:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574610.629957:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574610.629961:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488794240:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574611.141323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574611.589306:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574612.165332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574612.613390:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574613.189287:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574613.637400:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574614.213304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574614.661337:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574615.237376:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574615.685395:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574615.749361:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574615.749372:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574615.749376:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574615.749388:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5764 00000100:00080000:0.0:1788574615.749401:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574615.749409:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574615.749460:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488794368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574615.749493:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574615.749497:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574615.749498:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574615.749502:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5764 00000100:00080000:0.0:1788574615.749505:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574615.749509:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574615.749519:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574615.749532:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488794496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574615.749540:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574615.749543:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574615.749545:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574615.749547:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5764 00000100:00080000:0.0:1788574615.749551:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574615.749556:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574615.749563:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574615.749563:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488794368:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574615.749584:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488794624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574615.749600:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574615.749603:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574615.749605:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574615.749609:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5764 00000100:00080000:0.0:1788574615.749614:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574615.749620:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574615.749627:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574615.749642:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488794752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574615.749652:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574615.749656:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574615.749658:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574615.749663:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5764 00000100:00080000:0.0:1788574615.749669:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574615.749674:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574615.749680:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574615.749681:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488794496:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574615.749706:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488794880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574615.749728:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488794624:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574615.749760:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488794752:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574615.749803:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488794880:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574616.261310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574616.709358:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574617.285324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574617.733381:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574618.309270:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574618.757342:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788574618.821445:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:3.0:1788574618.821490:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788574618.821518:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:3.0:1788574618.821558:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:3.0:1788574618.821568:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:3.0:1788574618.821578:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:3.0:1788574618.821586:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:2.0:1788574618.821617:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574615/real 1788574618] req@ffff94d408d3ce00 x1875450488794880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574618.821645:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488794880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574618.821664:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574618.821669:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574618.821674:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488794880:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574618.821682:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574615/real 1788574618] req@ffff94d408d3d500 x1875450488794624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574618.821696:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488794624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574618.821705:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574618.821709:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574618.821713:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488794624:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574618.821721:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574615/real 1788574618] req@ffff94d408d3ca80 x1875450488794752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574618.821733:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488794752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574618.821741:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574618.821745:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574618.821748:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488794752:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574618.821789:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574615/real 1788574618] req@ffff94d408d3ea00 x1875450488794368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574618.821803:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488794368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574618.821812:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574618.821815:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574618.821819:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488794368:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574618.821826:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574615/real 1788574618] req@ffff94d408d3e680 x1875450488794496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574618.821838:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488794496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574618.821847:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574618.821851:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574618.821854:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488794496:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574619.333311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574619.781389:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574620.357309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574620.805388:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574620.869333:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574620.869341:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574620.869344:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574620.869354:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5769 00000100:00080000:3.0:1788574620.869364:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574620.869372:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574620.869408:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488795008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574620.869435:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574620.869438:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574620.869439:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574620.869442:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5769 00000100:00080000:3.0:1788574620.869445:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574620.869448:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574620.869456:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574620.869474:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488795136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574620.869483:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574620.869487:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788574620.869489:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488795008:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574620.869489:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574620.869502:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5769 00000100:00080000:3.0:1788574620.869506:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574620.869511:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574620.869516:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574620.869530:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488795264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574620.869540:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574620.869543:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574620.869545:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574620.869549:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5769 00000100:00080000:3.0:1788574620.869553:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788574620.869560:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574620.869561:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574620.869570:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574620.869584:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488795392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574620.869593:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574620/real 1788574620] req@ffff94d408c55880 x1875450488795008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574620.869604:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574620.869608:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1788574620.869611:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488795008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574620.869613:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574620.869623:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5769 00000100:00080000:2.0:1788574620.869627:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574620.869629:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574620.869633:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:3.0:1788574620.869638:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574620.869642:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488795008:192.168.202.125@tcp:250:kworker.0 00000008:00100000:3.0:1788574620.869646:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574620.869661:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488795520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574620.869674:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488795136:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574620.869708:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574620.869724:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574620/real 1788574620] req@ffff94d408c55180 x1875450488795136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574620.869756:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488795136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574620.869766:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574620.869770:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574620.869773:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488795136:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574620.869780:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488795264:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574620.869804:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574620.869817:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574620/real 1788574620] req@ffff94d408c54700 x1875450488795264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574620.869828:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488795264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574620.869844:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574620.869848:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574620.869851:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488795264:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574620.869858:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488795392:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574620.869877:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574620.869889:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574620/real 1788574620] req@ffff94d408c56300 x1875450488795392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574620.869916:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488795392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574620.869924:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574620.869927:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574620.869931:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488795392:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574620.869949:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488795520:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574620.869969:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574620.869981:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574620/real 1788574620] req@ffff94d408c55f80 x1875450488795520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574620.869993:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488795520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574620.870002:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574620.870005:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574620.870008:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488795520:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574621.381311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574621.829392:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574622.405285:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574622.853394:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574623.429290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574623.877367:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574624.453729:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574624.901331:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574625.477564:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574625.925380:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574625.989388:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574625.989400:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574625.989404:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574625.989415:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5774 00000100:00080000:3.0:1788574625.989429:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574625.989437:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574625.989483:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488795648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574625.989513:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574625.989516:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574625.989518:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574625.989521:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5774 00000100:00080000:3.0:1788574625.989523:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574625.989525:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574625.989534:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574625.989543:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488795776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574625.989551:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574625.989554:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574625.989555:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574625.989558:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5774 00000100:00080000:3.0:1788574625.989561:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574625.989563:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574625.989566:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574625.989574:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488795904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574625.989585:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574625.989587:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574625.989589:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574625.989592:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5774 00000100:00080000:3.0:1788574625.989593:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574625.989596:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574625.989599:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574625.989606:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488796032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574625.989613:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574625.989616:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574625.989617:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574625.989621:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5774 00000100:00080000:3.0:1788574625.989626:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788574625.989631:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488795648:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574625.989632:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574625.989643:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574625.989654:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488796160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574625.989730:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574625.989776:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574625/real 1788574625] req@ffff94d408c57480 x1875450488795648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574625.989795:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488795648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574625.989808:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574625.989811:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574625.989815:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488795648:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574625.989823:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488795776:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574625.989849:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574625.989863:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574625/real 1788574625] req@ffff94d408c54000 x1875450488795776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574625.989877:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488795776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574625.989884:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574625.989886:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574625.989888:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488795776:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574625.989893:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488795904:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574625.989909:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574625.989917:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574625/real 1788574625] req@ffff94d408c55f80 x1875450488795904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574625.989925:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488795904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574625.989931:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574625.989933:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574625.989935:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488795904:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574625.989957:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488796032:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574625.989969:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574625.989975:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574625/real 1788574625] req@ffff94d408c56300 x1875450488796032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574625.989983:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488796032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574625.989988:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574625.989990:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574625.989992:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488796032:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574625.989995:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488796160:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574625.990005:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574625.990010:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574625/real 1788574625] req@ffff94d408c54700 x1875450488796160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574625.990017:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488796160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574625.990022:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574625.990024:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574625.990025:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488796160:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574626.501497:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574626.949401:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574627.525596:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574627.973364:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574628.550753:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574628.997373:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574629.573426:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574630.021440:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574630.597334:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574631.045393:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574631.109363:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574631.109378:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574631.109386:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574631.109398:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5779 00000100:00080000:3.0:1788574631.109422:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574631.109432:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574631.109486:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488796288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574631.109525:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574631.109529:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574631.109531:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574631.109536:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5779 00000100:00080000:3.0:1788574631.109539:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574631.109543:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574631.109555:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574631.109569:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488796416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574631.109580:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574631.109584:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574631.109587:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574631.109592:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5779 00000100:00080000:3.0:1788574631.109596:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574631.109601:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574631.109606:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488796288:192.168.202.125@tcp:250:kworker.0 00000008:00100000:3.0:1788574631.109609:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574631.109626:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488796544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574631.109661:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574631.109665:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574631.109667:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574631.109672:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5779 00000100:00080000:3.0:1788574631.109692:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574631.109697:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574631.109703:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574631.109709:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574631.109721:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488796672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574631.109734:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574631.109738:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574631.109740:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574631.109745:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5779 00000100:00080000:3.0:1788574631.109751:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574631.109756:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574631.109761:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574631/real 1788574631] req@ffff94d408c55180 x1875450488796288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788574631.109764:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574631.109787:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488796288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574631.109788:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488796800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574631.109812:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574631.109817:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574631.109829:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488796288:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574631.109836:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488796416:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574631.109858:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574631.109868:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574631/real 1788574631] req@ffff94d408c55880 x1875450488796416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574631.109879:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488796416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574631.109885:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574631.109888:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574631.109892:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488796416:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574631.109916:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488796544:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574631.109928:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574631.109934:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574631/real 1788574631] req@ffff94d408c54380 x1875450488796544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574631.109942:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488796544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574631.109952:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574631.109955:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574631.109957:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488796544:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574631.109961:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488796672:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574631.109973:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574631.109980:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574631/real 1788574631] req@ffff94d408c57800 x1875450488796672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574631.109987:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488796672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574631.109999:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574631.110002:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574631.110004:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488796672:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574631.110011:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488796800:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574631.110023:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574631.110030:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574631/real 1788574631] req@ffff94d408c56a00 x1875450488796800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574631.110044:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488796800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574631.110052:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574631.110055:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574631.110059:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488796800:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574631.621334:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574632.069398:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574632.645356:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574633.093315:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574633.669330:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574634.117409:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574634.693291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574635.141420:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574635.717368:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574636.165362:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574636.229314:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574636.229322:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574636.229326:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574636.229339:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5784 00000100:00080000:3.0:1788574636.229350:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574636.229356:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574636.229389:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488796928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574636.229415:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574636.229419:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574636.229421:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574636.229426:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5784 00000100:00080000:3.0:1788574636.229429:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574636.229435:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574636.229445:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574636.229458:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488796928:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574636.229465:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488797056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574636.229487:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574636.229491:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574636.229494:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574636.229499:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5784 00000100:00080000:3.0:1788574636.229510:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574636.229514:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:2.0:1788574636.229519:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:3.0:1788574636.229521:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574636.229534:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488797184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574636.229542:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574636.229545:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574636.229546:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574636.229550:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5784 00000100:00100000:2.0:1788574636.229554:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574636/real 1788574636] req@ffff94d408c57100 x1875450488796928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574636.229554:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574636.229560:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574636.229567:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574636.229580:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488796928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574636.229592:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488797312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574636.229624:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574636.229626:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574636.229630:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:3.0:1788574636.229635:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1788574636.229636:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488796928:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574636.229639:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574636.229647:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5784 00000100:00080000:3.0:1788574636.229652:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574636.229657:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574636.229660:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488797056:192.168.202.125@tcp:38:kworker.0 00000008:00100000:3.0:1788574636.229665:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574636.229683:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488797440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574636.229686:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574636.229700:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574636/real 1788574636] req@ffff94d408c55c00 x1875450488797056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574636.229710:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488797056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574636.229717:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574636.229720:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574636.229723:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488797056:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574636.229728:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488797184:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574636.229739:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574636.229745:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574636/real 1788574636] req@ffff94d408c54e00 x1875450488797184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574636.229753:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488797184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574636.229758:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574636.229760:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574636.229761:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488797184:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574636.229765:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488797312:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574636.229775:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574636.229780:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574636/real 1788574636] req@ffff94d408c56d80 x1875450488797312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574636.229788:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488797312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574636.229793:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574636.229794:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574636.229796:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488797312:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574636.229806:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488797440:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574636.229815:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574636.229820:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574636/real 1788574636] req@ffff94d408c54a80 x1875450488797440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574636.229827:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488797440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574636.229832:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574636.229834:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574636.229836:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488797440:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574636.741328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574637.189367:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574637.765319:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574638.213381:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574638.789295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574639.237365:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574639.813292:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574640.261410:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574640.837326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574641.285395:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574641.349281:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574641.349293:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574641.349297:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574641.349306:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5789 00000100:00080000:3.0:1788574641.349316:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574641.349322:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574641.349358:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488797568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574641.349381:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574641.349384:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574641.349386:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574641.349389:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5789 00000100:00080000:3.0:1788574641.349391:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574641.349394:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574641.349402:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574641.349413:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488797696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574641.349417:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488797568:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574641.349424:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574641.349427:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574641.349428:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574641.349432:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5789 00000100:00080000:3.0:1788574641.349435:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574641.349438:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574641.349441:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574641.349450:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488797824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574641.349457:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574641.349459:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574641.349462:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574641.349466:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5789 00000100:00080000:3.0:1788574641.349470:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574641.349473:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574641.349477:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574641.349486:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488797952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574641.349496:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574641.349501:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574641.349507:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574641.349509:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574641.349515:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5789 00000100:00080000:3.0:1788574641.349520:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574641.349525:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574641.349547:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574641.349558:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574641/real 1788574641] req@ffff94d408c56680 x1875450488797568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574641.349563:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488798080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574641.349584:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488797568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574641.349593:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574641.349596:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574641.349599:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488797568:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788574641.349621:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488797696:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574641.349648:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574641.349662:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574641/real 1788574641] req@ffff94d408c57b80 x1875450488797696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574641.349676:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488797696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574641.349684:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574641.349686:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574641.349688:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488797696:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574641.349693:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488797824:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574641.349709:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574641.349717:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574641/real 1788574641] req@ffff94d408c54a80 x1875450488797824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574641.349725:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488797824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574641.349731:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574641.349734:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574641.349736:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488797824:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574641.349739:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488797952:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574641.349749:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574641.349755:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574641/real 1788574641] req@ffff94d408c56d80 x1875450488797952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574641.349767:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488797952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574641.349772:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574641.349774:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574641.349776:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488797952:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574641.349779:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488798080:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574641.349788:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574641.349793:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574641/real 1788574641] req@ffff94d408c54e00 x1875450488798080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574641.349801:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488798080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574641.349806:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574641.349808:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574641.349810:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488798080:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574641.861307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574642.309431:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574642.885346:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574643.333359:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574643.909328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574644.357433:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574644.933304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574645.381360:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574645.957307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574646.405369:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574646.469358:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574646.469371:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574646.469376:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574646.469388:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5794 00000100:00080000:0.0:1788574646.469410:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574646.469420:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574646.469467:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f800 x1875450488798208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574646.469514:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574646.469518:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574646.469519:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574646.469523:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5794 00000100:00080000:0.0:1788574646.469526:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574646.469537:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574646.469549:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574646.469568:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f100 x1875450488798336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574646.469574:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488798208:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788574646.469596:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574646.469600:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574646.469602:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574646.469607:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5794 00000100:00080000:0.0:1788574646.469611:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574646.469616:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574646.469622:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574646.469636:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488798464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574646.469645:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574646.469648:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574646.469655:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788574646.469655:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788574646.469668:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5794 00000100:00080000:0.0:1788574646.469676:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574646.469682:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574646.469687:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574646.469697:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488798592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574646.469706:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574646.469707:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574646/real 1788574646] req@ffff94d408d3f800 x1875450488798208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574646.469720:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574646.469724:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574646.469741:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5794 00000100:00080000:0.0:1788574646.469747:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574646.469750:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574646.469754:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f800 x1875450488798208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788574646.469755:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574646.469774:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488798720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574646.469776:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574646.469794:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574646.469807:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488798208:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788574646.469842:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488798336:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574646.469861:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574646.469870:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574646/real 1788574646] req@ffff94d408d3f100 x1875450488798336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574646.469878:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f100 x1875450488798336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574646.469884:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574646.469886:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574646.469889:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488798336:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574646.469893:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488798464:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574646.469903:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574646.469909:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574646/real 1788574646] req@ffff94d408d3e680 x1875450488798464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574646.469921:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488798464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574646.469927:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574646.469928:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574646.469930:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488798464:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574646.469934:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488798592:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574646.469945:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574646.469951:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574646/real 1788574646] req@ffff94d408d3ea00 x1875450488798592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574646.469958:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488798592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574646.469963:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574646.469972:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574646.469974:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488798592:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574646.469978:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488798720:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574646.469989:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574646.469996:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574646/real 1788574646] req@ffff94d408d3ca80 x1875450488798720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574646.470003:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488798720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574646.470013:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574646.470015:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574646.470017:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488798720:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574646.981308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574647.429438:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574648.005339:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574648.453368:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574649.029351:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574649.477432:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574650.053307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574650.501347:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574651.077333:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574651.525391:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574651.589382:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574651.589398:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574651.589404:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574651.589418:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5799 00000100:00080000:0.0:1788574651.589446:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574651.589457:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574651.589506:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488798848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574651.589536:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574651.589539:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574651.589541:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574651.589544:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5799 00000100:00080000:0.0:1788574651.589547:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574651.589550:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574651.589559:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574651.589571:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488798976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574651.589579:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574651.589582:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574651.589584:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574651.589586:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5799 00000100:00080000:0.0:1788574651.589589:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574651.589591:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574651.589594:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574651.589604:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488799104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574651.589604:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488798848:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788574651.589624:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574651.589626:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574651.589627:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574651.589638:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5799 00000100:00080000:0.0:1788574651.589643:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574651.589646:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574651.589650:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574651.589659:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488799232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574651.589666:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574651.589669:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574651.589670:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574651.589673:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5799 00000100:00080000:0.0:1788574651.589675:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574651.589677:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574651.589681:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574651.589688:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488799360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574651.589696:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574651.589746:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574651/real 1788574651] req@ffff94d408d3d500 x1875450488798848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574651.589772:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488798848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574651.589786:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574651.589792:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574651.589798:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488798848:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788574651.589807:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488798976:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574651.589830:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574651.589843:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574651/real 1788574651] req@ffff94d408d3ce00 x1875450488798976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574651.589857:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488798976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574651.589866:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574651.589870:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574651.589874:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488798976:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574651.589903:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488799104:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574651.589922:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574651.589933:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574651/real 1788574651] req@ffff94d408d3df80 x1875450488799104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574651.589947:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488799104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574651.589957:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574651.589961:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574651.589965:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488799104:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574651.589972:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488799232:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574651.589991:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574651.590001:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574651/real 1788574651] req@ffff94d408d3d180 x1875450488799232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574651.590012:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488799232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574651.590020:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574651.590024:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574651.590027:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488799232:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574651.590033:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488799360:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574651.590054:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574651.590065:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574651/real 1788574651] req@ffff94d408d3c000 x1875450488799360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574651.590077:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488799360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574651.590086:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574651.590090:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574651.590101:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488799360:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574652.101313:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574652.549496:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574653.125346:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574653.573446:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574654.149333:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574654.597355:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574655.173296:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574655.621332:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574656.197321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574656.645464:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574656.709373:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574656.709396:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574656.709400:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574656.709409:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5805 00000100:00080000:3.0:1788574656.709420:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574656.709428:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574656.709467:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488799488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574656.709494:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574656.709497:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574656.709499:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574656.709501:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5805 00000100:00080000:3.0:1788574656.709504:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574656.709507:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574656.709516:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574656.709526:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488799616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574656.709535:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574656.709541:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488799488:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574656.709541:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574656.709543:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574656.709551:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5805 00000100:00080000:3.0:1788574656.709554:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574656.709556:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574656.709559:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574656.709568:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488799744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574656.709575:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574656.709578:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574656.709579:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574656.709582:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5805 00000100:00080000:3.0:1788574656.709585:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574656.709587:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574656.709590:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574656.709598:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488799872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574656.709605:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788574656.709606:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574656.709610:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574656.709612:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574656.709616:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5805 00000100:00080000:3.0:1788574656.709619:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574656.709622:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574656.709626:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574656.709634:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488800000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574656.709643:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574656/real 1788574656] req@ffff94d408c55c00 x1875450488799488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574656.709672:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488799488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574656.709681:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574656.709684:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574656.709687:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488799488:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788574656.709706:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488799616:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574656.709722:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574656.709730:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574656/real 1788574656] req@ffff94d408c57100 x1875450488799616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574656.709739:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488799616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574656.709744:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574656.709747:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574656.709749:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488799616:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574656.709752:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488799744:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574656.709763:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574656.709768:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574656/real 1788574656] req@ffff94d408c56a00 x1875450488799744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574656.709776:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488799744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574656.709786:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574656.709788:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574656.709790:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488799744:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574656.709794:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488799872:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574656.709803:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574656.709809:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574656/real 1788574656] req@ffff94d408c57800 x1875450488799872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574656.709816:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488799872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574656.709821:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574656.709823:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574656.709825:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488799872:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574656.709828:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488800000:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574656.709837:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574656.709842:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574656/real 1788574656] req@ffff94d408c54380 x1875450488800000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574656.709849:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488800000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574656.709854:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574656.709856:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574656.709859:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488800000:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574657.221291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574657.669419:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574658.245376:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574658.693436:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574659.269277:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574659.717438:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574660.293315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574660.741501:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574661.317320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574661.765403:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574661.829345:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574661.829355:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574661.829360:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574661.829370:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5810 00000100:00080000:3.0:1788574661.829384:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574661.829394:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574661.829445:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488800128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574661.829479:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574661.829484:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574661.829487:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574661.829492:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5810 00000100:00080000:3.0:1788574661.829496:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574661.829501:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574661.829514:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574661.829528:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488800256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574661.829537:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488800128:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574661.829539:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574661.829547:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574661.829548:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574661.829552:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5810 00000100:00080000:3.0:1788574661.829555:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574661.829558:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574661.829562:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574661.829572:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488800384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574661.829579:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574661.829582:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574661.829584:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574661.829587:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5810 00000100:00080000:3.0:1788574661.829590:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574661.829592:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574661.829596:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574661.829603:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488800512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574661.829609:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574661.829611:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574661.829613:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574661.829616:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5810 00000100:00080000:3.0:1788574661.829619:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574661.829621:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574661.829624:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788574661.829629:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574661.829633:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488800640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574661.829670:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574661/real 1788574661] req@ffff94d408c55880 x1875450488800128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574661.829687:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488800128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574661.829696:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574661.829700:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574661.829703:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488800128:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788574661.829727:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488800256:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574661.829744:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574661.829752:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574661/real 1788574661] req@ffff94d408c55180 x1875450488800256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574661.829760:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488800256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574661.829771:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574661.829774:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574661.829776:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488800256:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574661.829779:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488800384:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574661.829790:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574661.829795:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574661/real 1788574661] req@ffff94d408c54700 x1875450488800384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574661.829803:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488800384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574661.829809:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574661.829811:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574661.829812:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488800384:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574661.829816:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488800512:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574661.829827:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574661.829832:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574661/real 1788574661] req@ffff94d408c56300 x1875450488800512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574661.829840:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488800512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574661.829845:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574661.829846:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574661.829849:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488800512:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574661.829853:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488800640:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574661.829863:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574661.829868:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574661/real 1788574661] req@ffff94d408c55f80 x1875450488800640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574661.829877:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488800640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574661.829885:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574661.829888:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574661.829892:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488800640:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574662.341354:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574662.789374:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574663.365290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574663.813385:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574664.389388:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574664.837374:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574665.413322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574665.861483:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574666.437303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574666.885437:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574666.949310:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574666.949318:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574666.949322:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574666.949330:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5815 00000100:00080000:3.0:1788574666.949340:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574666.949345:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574666.949378:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488800768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574666.949404:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574666.949407:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574666.949408:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574666.949411:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5815 00000100:00080000:3.0:1788574666.949413:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574666.949416:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574666.949423:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574666.949431:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488800896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574666.949439:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574666.949441:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574666.949443:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574666.949445:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5815 00000100:00080000:3.0:1788574666.949447:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574666.949450:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574666.949452:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574666.949461:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488801024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574666.949468:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488800768:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574666.949471:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574666.949476:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574666.949480:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574666.949483:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5815 00000100:00080000:3.0:1788574666.949485:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574666.949488:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574666.949491:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574666.949499:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488801152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574666.949505:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574666.949508:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574666.949509:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574666.949511:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5815 00000100:00080000:3.0:1788574666.949514:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574666.949516:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574666.949519:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574666.949526:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488801280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574666.949535:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574666.949564:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574666/real 1788574666] req@ffff94d408c54000 x1875450488800768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574666.949578:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488800768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574666.949587:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574666.949589:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574666.949592:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488800768:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788574666.949597:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488800896:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574666.949611:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574666.949618:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574666/real 1788574666] req@ffff94d408c57480 x1875450488800896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574666.949626:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488800896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574666.949631:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574666.949633:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574666.949635:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488800896:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574666.949653:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488801024:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574666.949663:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574666.949669:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574666/real 1788574666] req@ffff94d408c55f80 x1875450488801024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574666.949676:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488801024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574666.949685:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574666.949688:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574666.949691:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488801024:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574666.949698:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488801152:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574666.949717:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574666.949728:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574666/real 1788574666] req@ffff94d408c56300 x1875450488801152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574666.949740:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488801152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574666.949746:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574666.949748:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574666.949750:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488801152:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574666.949754:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488801280:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574666.949767:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574666.949774:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574666/real 1788574666] req@ffff94d408c54700 x1875450488801280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574666.949781:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488801280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574666.949786:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574666.949788:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574666.949790:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488801280:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574667.461324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574667.909420:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574668.485342:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574668.933416:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574669.509395:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574669.957435:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574670.533381:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574670.981396:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574671.557345:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574672.005331:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574672.069296:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574672.069307:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574672.069312:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574672.069321:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5820 00000100:00080000:0.0:1788574672.069331:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574672.069337:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574672.069367:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488801408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574672.069391:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574672.069395:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574672.069398:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574672.069402:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5820 00000100:00080000:0.0:1788574672.069405:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574672.069410:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574672.069419:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574672.069432:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488801536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574672.069438:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488801408:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788574672.069443:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574672.069446:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574672.069448:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574672.069451:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5820 00000100:00080000:0.0:1788574672.069455:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574672.069458:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574672.069462:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574672.069471:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488801664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574672.069478:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574672.069482:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574672.069483:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574672.069486:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5820 00000100:00080000:0.0:1788574672.069489:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574672.069493:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:1.0:1788574672.069496:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:0.0:1788574672.069497:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574672.069505:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488801792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574672.069514:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574672.069517:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574672.069519:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574672.069522:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5820 00000100:00080000:0.0:1788574672.069525:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574672.069528:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574672.069532:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574672/real 1788574672] req@ffff94d408d3c380 x1875450488801408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788574672.069533:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574672.069542:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488801920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574672.069550:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488801408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574672.069559:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574672.069563:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574672.069567:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488801408:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788574672.069590:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488801536:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574672.069609:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574672.069617:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574672/real 1788574672] req@ffff94d408d3f480 x1875450488801536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574672.069626:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488801536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574672.069631:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574672.069633:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574672.069636:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488801536:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574672.069640:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488801664:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574672.069650:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574672.069656:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574672/real 1788574672] req@ffff94d408d3c700 x1875450488801664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574672.069664:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488801664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574672.069669:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574672.069671:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574672.069673:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488801664:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574672.069676:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488801792:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574672.069686:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574672.069691:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574672/real 1788574672] req@ffff94d408d3e300 x1875450488801792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574672.069698:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488801792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574672.069703:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574672.069705:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574672.069715:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488801792:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574672.069718:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488801920:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574672.069730:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574672.069736:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574672/real 1788574672] req@ffff94d408d3dc00 x1875450488801920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574672.069748:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488801920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574672.069753:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574672.069755:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574672.069757:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488801920:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574672.581277:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574673.029400:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574673.605324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574674.053425:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574674.629301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574675.077320:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574675.653292:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574676.101506:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574676.677303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574677.125367:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574677.189300:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574677.189310:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574677.189315:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574677.189327:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5825 00000100:00080000:0.0:1788574677.189339:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574677.189347:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574677.189384:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488802048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574677.189413:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574677.189416:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574677.189418:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574677.189421:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5825 00000100:00080000:0.0:1788574677.189425:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574677.189428:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574677.189437:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574677.189449:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ed80 x1875450488802176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574677.189458:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574677.189461:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488802048:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788574677.189462:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574677.189467:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574677.189476:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5825 00000100:00080000:0.0:1788574677.189480:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574677.189486:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574677.189491:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574677.189516:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488802304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574677.189531:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574677.189535:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574677.189536:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574677.189540:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5825 00000100:00080000:0.0:1788574677.189544:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574677.189548:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574677.189553:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574677.189564:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488802432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574677.189571:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574677.189573:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574677.189574:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574677.189576:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5825 00000100:00080000:0.0:1788574677.189578:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574677.189580:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574677.189582:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574677.189588:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488802560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574677.189602:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574677.189640:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574677/real 1788574677] req@ffff94d408d3d880 x1875450488802048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574677.189786:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488802048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574677.189805:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574677.189810:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574677.189816:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488802048:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574677.189901:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488802176:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574677.189937:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574677.189949:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574677/real 1788574677] req@ffff94d408d3ed80 x1875450488802176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574677.189959:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ed80 x1875450488802176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574677.189965:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574677.189967:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574677.189970:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488802176:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574677.189974:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488802304:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574677.189987:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574677.189994:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574677/real 1788574677] req@ffff94d408d3dc00 x1875450488802304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574677.190001:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488802304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574677.190008:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574677.190010:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574677.190012:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488802304:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574677.190016:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488802432:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574677.190027:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574677.190033:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574677/real 1788574677] req@ffff94d408d3e300 x1875450488802432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574677.190045:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488802432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574677.190054:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574677.190057:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574677.190061:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488802432:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574677.190068:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488802560:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574677.190089:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574677.190101:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574677/real 1788574677] req@ffff94d408d3c700 x1875450488802560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574677.190113:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488802560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574677.190122:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574677.190126:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574677.190130:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488802560:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574677.701306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574678.149394:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574678.725314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574679.173408:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574679.749356:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574680.197433:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574680.773337:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574681.221396:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574681.797357:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574682.245388:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574682.309307:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574682.309315:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574682.309319:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574682.309327:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5830 00000100:00080000:1.0:1788574682.309335:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574682.309342:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574682.309374:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488802688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574682.309402:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574682.309405:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574682.309406:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574682.309409:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5830 00000100:00080000:1.0:1788574682.309411:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574682.309413:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574682.309419:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574682.309427:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488802816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574682.309434:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574682.309437:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574682.309438:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574682.309441:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5830 00000100:00080000:1.0:1788574682.309443:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574682.309445:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574682.309447:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574682.309456:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488802944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574682.309463:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488802688:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788574682.309465:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574682.309472:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574682.309474:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574682.309477:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5830 00000100:00080000:1.0:1788574682.309479:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574682.309481:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574682.309485:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574682.309493:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488803072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574682.309499:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574682.309501:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574682.309502:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574682.309505:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5830 00000100:00080000:1.0:1788574682.309507:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574682.309509:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574682.309512:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574682.309520:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488803200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574682.309556:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488802816:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574682.309587:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488802944:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574682.309608:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488803072:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574682.309630:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488803200:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574682.821371:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574683.269401:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574683.845336:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574684.293418:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574684.869311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574685.317350:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788574685.381593:0:2392:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:1.0:1788574685.381654:0:2392:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788574685.381671:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788574685.381720:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788574685.381731:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788574685.381742:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788574685.381756:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:3.0:1788574685.381793:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574682/real 1788574685] req@ffff94d4049bc380 x1875450488802944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574685.381821:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc380 x1875450488802944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574685.381834:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574685.381840:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574685.381845:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488802944:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574685.381853:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574682/real 1788574685] req@ffff94d4049bf480 x1875450488803072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574685.381867:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488803072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574685.381875:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574685.381879:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574685.381883:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488803072:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574685.381890:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574682/real 1788574685] req@ffff94d4049bea00 x1875450488803200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574685.381911:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488803200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574685.381925:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574685.381928:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574685.381932:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488803200:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574685.381939:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574682/real 1788574685] req@ffff94d4049bce00 x1875450488802688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574685.381952:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488802688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574685.381959:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574685.381963:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574685.381966:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488802688:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574685.381974:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574682/real 1788574685] req@ffff94d4049bd180 x1875450488802816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574685.381985:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488802816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574685.381992:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574685.381995:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574685.381998:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488802816:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574685.893318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574686.341370:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574686.917287:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574687.365366:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574687.429366:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574687.429377:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574687.429381:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574687.429392:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5835 00000100:00080000:0.0:1788574687.429413:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574687.429421:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574687.429457:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488803328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574687.429488:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574687.429495:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574687.429497:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574687.429501:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5835 00000100:00080000:0.0:1788574687.429506:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574687.429511:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574687.429521:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574687.429532:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488803328:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574687.429540:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488803456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574687.429555:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574687.429559:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574687.429561:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574687.429566:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5835 00000100:00080000:0.0:1788574687.429580:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574687.429585:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574687.429590:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574687.429607:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488803584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574687.429610:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788574687.429623:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574687.429627:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574687.429630:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574687.429636:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5835 00000100:00080000:0.0:1788574687.429640:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574687.429645:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574687.429653:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574687.429655:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574687/real 1788574687] req@ffff94d408d3f480 x1875450488803328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574687.429669:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488803712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574687.429683:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574687.429683:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488803328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574687.429693:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574687.429698:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574687.429709:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5835 00000100:00080000:3.0:1788574687.429711:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574687.429714:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574687.429720:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:0.0:1788574687.429725:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574687.429731:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488803328:192.168.202.125@tcp:250:kworker.0 00000008:00100000:0.0:1788574687.429734:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574687.429752:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488803840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574687.429764:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488803456:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574687.429795:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574687.429810:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574687/real 1788574687] req@ffff94d408d3c380 x1875450488803456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574687.429841:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488803456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574687.429850:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574687.429854:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574687.429857:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488803456:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574687.429866:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488803584:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574687.429889:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574687.429908:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574687/real 1788574687] req@ffff94d408d3c000 x1875450488803584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574687.429926:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488803584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574687.429934:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574687.429937:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574687.429940:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488803584:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574687.429946:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488803712:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574687.429966:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574687.429977:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574687/real 1788574687] req@ffff94d408d3d180 x1875450488803712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574687.429989:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488803712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574687.429996:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574687.429999:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574687.430002:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488803712:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574687.430018:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488803840:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574687.430038:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574687.430050:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574687/real 1788574687] req@ffff94d408d3df80 x1875450488803840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574687.430147:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488803840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574687.430161:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574687.430165:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574687.430168:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488803840:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574687.941269:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574688.389394:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574688.965274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574689.413311:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574689.989276:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574690.437352:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574691.013434:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574691.461321:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574692.037616:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574692.485332:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574692.549348:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574692.549396:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574692.549401:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574692.549412:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5840 00000100:00080000:3.0:1788574692.549423:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574692.549431:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574692.549467:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488803968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574692.549500:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574692.549504:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574692.549506:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574692.549510:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5840 00000100:00080000:3.0:1788574692.549513:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574692.549518:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574692.549527:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574692.549544:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488803968:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574692.549545:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488804096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574692.549563:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574692.549567:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574692.549570:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574692.549574:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5840 00000100:00080000:3.0:1788574692.549578:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574692.549583:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574692.549589:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574692.549600:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488804224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574692.549607:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574692.549616:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574692.549621:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574692.549623:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574692.549629:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5840 00000100:00080000:3.0:1788574692.549634:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574692.549640:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574692/real 1788574692] req@ffff94d408c55180 x1875450488803968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574692.549641:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574692.549652:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574692.549660:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488803968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574692.549668:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488804352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574692.549679:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574692.549680:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574692.549686:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574692.549689:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:3.0:1788574692.549691:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788574692.549695:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488803968:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574692.549698:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5840 00000100:00080000:3.0:1788574692.549705:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574692.549709:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574692.549714:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574692.549720:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488804096:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574692.549727:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488804480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574692.549742:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574692.549753:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574692/real 1788574692] req@ffff94d408c55880 x1875450488804096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574692.549768:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488804096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574692.549774:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574692.549776:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574692.549778:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488804096:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574692.549782:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488804224:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574692.549792:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574692.549797:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574692/real 1788574692] req@ffff94d408c54380 x1875450488804224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574692.549805:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488804224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574692.549810:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574692.549812:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574692.549814:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488804224:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574692.549817:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488804352:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574692.549827:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574692.549833:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574692/real 1788574692] req@ffff94d408c57800 x1875450488804352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574692.549840:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488804352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574692.549850:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574692.549852:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574692.549854:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488804352:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574692.549862:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488804480:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574692.549872:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574692.549877:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574692/real 1788574692] req@ffff94d408c56a00 x1875450488804480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574692.549884:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488804480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574692.549889:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574692.549890:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574692.549892:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488804480:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574693.061880:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574693.509335:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574694.085500:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574694.533373:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574695.109476:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574695.557385:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574696.133983:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574696.581409:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574697.157300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574697.605354:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574697.669320:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574697.669330:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574697.669333:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574697.669342:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5845 00000100:00080000:1.0:1788574697.669352:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574697.669358:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574697.669390:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488804608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574697.669413:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574697.669415:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574697.669417:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574697.669420:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5845 00000100:00080000:1.0:1788574697.669422:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574697.669424:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574697.669431:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574697.669440:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488804736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574697.669447:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574697.669450:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574697.669451:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574697.669454:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5845 00000100:00080000:1.0:1788574697.669457:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574697.669460:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488804608:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788574697.669461:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574697.669467:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574697.669475:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488804864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574697.669481:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574697.669483:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574697.669485:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574697.669488:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5845 00000100:00080000:1.0:1788574697.669491:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574697.669493:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574697.669497:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574697.669505:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488804992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574697.669511:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574697.669513:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574697.669514:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574697.669516:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5845 00000100:00080000:1.0:1788574697.669519:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574697.669522:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574697.669525:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574697.669533:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488805120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574697.669534:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574697.669579:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574697/real 1788574697] req@ffff94d4049bfb80 x1875450488804608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574697.669601:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488804608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574697.669612:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574697.669616:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574697.669621:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488804608:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574697.669646:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488804736:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574697.669671:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574697.669685:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574697/real 1788574697] req@ffff94d4049bc000 x1875450488804736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574697.669705:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488804736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574697.669714:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574697.669718:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574697.669722:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488804736:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574697.669728:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488804864:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574697.669749:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574697.669761:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574697/real 1788574697] req@ffff94d4049bc700 x1875450488804864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574697.669774:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488804864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574697.669783:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574697.669786:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574697.669789:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488804864:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574697.669796:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488804992:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574697.669818:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574697.669830:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574697/real 1788574697] req@ffff94d4049bed80 x1875450488804992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574697.669842:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488804992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574697.669857:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574697.669860:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574697.669864:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488804992:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574697.669870:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488805120:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574697.669889:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574697.669930:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574697/real 1788574697] req@ffff94d4049bf800 x1875450488805120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574697.669961:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488805120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574697.669971:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574697.669975:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574697.669979:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488805120:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574698.181255:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574698.629316:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574699.205315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574699.653454:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574700.229331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574700.677405:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574701.253304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574701.701481:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574702.277265:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574702.725345:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574702.789322:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574702.789333:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574702.789337:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574702.789347:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5851 00000100:00080000:3.0:1788574702.789358:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574702.789366:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574702.789406:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488805248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574702.789432:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574702.789452:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574702.789454:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574702.789458:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5851 00000100:00080000:3.0:1788574702.789463:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574702.789469:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574702.789479:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574702.789484:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488805248:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574702.789496:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488805376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574702.789509:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574702.789513:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574702.789515:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574702.789520:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5851 00000100:00080000:3.0:1788574702.789525:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574702.789529:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574702.789535:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574702.789546:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488805504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574702.789554:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788574702.789555:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574702.789560:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574702.789562:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574702.789566:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5851 00000100:00080000:3.0:1788574702.789570:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574702.789573:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574702.789577:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574702.789586:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488805632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574702.789594:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574702/real 1788574702] req@ffff94d408c57100 x1875450488805248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574702.789595:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574702.789602:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574702.789604:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574702.789612:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5851 00000100:00080000:3.0:1788574702.789616:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574702.789619:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574702.789623:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574702.789625:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488805248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574702.789632:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488805760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574702.789639:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574702.789644:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574702.789648:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488805248:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788574702.789667:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488805376:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574702.789682:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574702.789690:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574702/real 1788574702] req@ffff94d408c55c00 x1875450488805376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574702.789698:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488805376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574702.789704:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574702.789706:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574702.789708:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488805376:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574702.789711:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488805504:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574702.789721:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574702.789727:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574702/real 1788574702] req@ffff94d408c54e00 x1875450488805504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574702.789734:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488805504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574702.789739:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574702.789741:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574702.789743:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488805504:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574702.789747:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488805632:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574702.789757:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574702.789763:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574702/real 1788574702] req@ffff94d408c56d80 x1875450488805632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574702.789770:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488805632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574702.789786:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574702.789787:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574702.789789:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488805632:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574702.789792:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488805760:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574702.789801:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574702.789806:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574702/real 1788574702] req@ffff94d408c54a80 x1875450488805760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574702.789813:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488805760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574702.789818:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574702.789820:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574702.789821:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488805760:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574703.301285:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574703.749383:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574704.325271:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574704.773347:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574705.349285:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574705.797332:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574706.373285:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574706.821325:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574707.397280:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574707.845329:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574707.909298:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574707.909311:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574707.909315:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574707.909326:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5856 00000100:00080000:3.0:1788574707.909337:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574707.909346:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574707.909385:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488805888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574707.909416:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574707.909421:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574707.909423:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574707.909432:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5856 00000100:00080000:3.0:1788574707.909436:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574707.909441:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574707.909450:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574707.909466:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488805888:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574707.909468:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488806016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574707.909485:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574707.909489:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574707.909491:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574707.909495:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5856 00000100:00080000:3.0:1788574707.909499:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574707.909504:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574707.909509:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574707.909523:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488806144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574707.909530:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574707.909541:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574707.909545:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574707.909547:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574707.909552:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5856 00000100:00080000:3.0:1788574707.909558:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574707.909563:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574707/real 1788574707] req@ffff94d408c57b80 x1875450488805888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574707.909565:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574707.909575:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574707.909583:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488805888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574707.909592:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488806272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574707.909593:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574707.909600:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574707.909608:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488805888:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574707.909615:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574707.909620:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574707.909622:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574707.909627:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5856 00000100:00100000:1.0:1788574707.909632:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488806016:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788574707.909632:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574707.909642:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574707.909649:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788574707.909654:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574707.909665:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488806400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574707.909666:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574707/real 1788574707] req@ffff94d408c56680 x1875450488806016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574707.909696:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488806016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574707.909704:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574707.909707:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574707.909710:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488806016:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574707.909714:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488806144:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574707.909727:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574707.909733:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574707/real 1788574707] req@ffff94d408c54a80 x1875450488806144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574707.909746:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488806144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574707.909751:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574707.909753:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574707.909755:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488806144:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574707.909758:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488806272:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574707.909773:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574707.909784:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574707/real 1788574707] req@ffff94d408c56d80 x1875450488806272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574707.909797:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488806272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574707.909805:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574707.909809:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574707.909812:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488806272:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574707.909828:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488806400:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574707.909867:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574707.909880:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574707/real 1788574707] req@ffff94d408c54e00 x1875450488806400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574707.909892:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488806400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574707.909909:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574707.909913:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574707.909916:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488806400:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574708.421257:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574708.869387:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574709.445291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574709.893339:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574710.469276:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574710.917342:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574711.493303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574711.941353:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574712.517296:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574712.965373:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574713.029456:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574713.029467:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574713.029474:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574713.029488:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5861 00000100:00080000:3.0:1788574713.029502:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574713.029526:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574713.029577:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488806528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574713.029609:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574713.029613:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574713.029614:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574713.029618:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5861 00000100:00080000:3.0:1788574713.029622:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574713.029626:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574713.029637:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574713.029649:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488806656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574713.029660:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488806528:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574713.029660:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574713.029670:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574713.029673:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574713.029678:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5861 00000100:00080000:3.0:1788574713.029682:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574713.029685:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574713.029690:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574713.029699:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488806784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574713.029706:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574713.029708:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574713.029710:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574713.029714:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5861 00000100:00080000:3.0:1788574713.029717:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574713.029720:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574713.029723:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574713.029731:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488806912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574713.029737:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574713.029740:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574713.029741:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574713.029748:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5861 00000800:00000100:1.0:1788574713.029749:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574713.029755:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574713.029759:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574713.029787:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574713/real 1788574713] req@ffff94d408c55c00 x1875450488806528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574713.029803:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488806528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574713.029813:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574713.029817:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574713.029821:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488806528:192.168.202.125@tcp:250:kworker.0 00000008:00100000:3.0:1788574713.029827:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574713.029840:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488807040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574713.029850:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488806656:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574713.029879:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574713.029892:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574713/real 1788574713] req@ffff94d408c57100 x1875450488806656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574713.029902:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488806656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574713.029913:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574713.029917:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574713.029920:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488806656:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574713.029925:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488806784:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574713.029942:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574713.029954:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574713/real 1788574713] req@ffff94d408c56a00 x1875450488806784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574713.029966:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488806784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574713.029974:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574713.029977:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574713.029979:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488806784:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574713.029984:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488806912:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574713.030000:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574713.030009:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574713/real 1788574713] req@ffff94d408c57800 x1875450488806912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574713.030018:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488806912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574713.030024:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574713.030026:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574713.030029:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488806912:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574713.030042:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488807040:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574713.030055:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574713.030062:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574713/real 1788574713] req@ffff94d408c54380 x1875450488807040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574713.030070:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488807040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574713.030075:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574713.030077:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574713.030079:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488807040:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574713.541284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574713.989335:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574714.565267:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574715.013427:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574715.589258:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574716.037355:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574716.613364:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574717.061354:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574717.637269:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574718.085336:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574718.149282:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574718.149290:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574718.149293:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574718.149301:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5866 00000100:00080000:3.0:1788574718.149310:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574718.149316:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574718.149349:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488807168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574718.149372:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574718.149376:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574718.149377:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574718.149380:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5866 00000100:00080000:3.0:1788574718.149382:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574718.149384:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574718.149391:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574718.149399:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488807296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574718.149408:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574718.149410:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574718.149411:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574718.149415:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5866 00000100:00080000:3.0:1788574718.149417:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574718.149419:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574718.149423:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574718.149431:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488807168:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574718.149432:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488807424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574718.149445:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574718.149447:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574718.149448:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574718.149451:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5866 00000100:00080000:3.0:1788574718.149454:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574718.149456:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574718.149459:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574718.149465:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488807552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574718.149471:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574718.149474:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574718.149475:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574718.149478:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5866 00000100:00080000:3.0:1788574718.149480:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574718.149482:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574718.149485:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574718.149492:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488807680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574718.149499:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574718.149531:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574718/real 1788574718] req@ffff94d408c55880 x1875450488807168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574718.149545:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488807168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574718.149553:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574718.149556:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574718.149559:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488807168:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788574718.149566:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488807296:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574718.149585:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574718.149597:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574718/real 1788574718] req@ffff94d408c55180 x1875450488807296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574718.149611:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488807296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574718.149620:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574718.149623:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574718.149636:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488807296:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574718.149656:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488807424:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574718.149672:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574718.149679:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574718/real 1788574718] req@ffff94d408c54700 x1875450488807424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574718.149687:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488807424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574718.149693:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574718.149695:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574718.149697:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488807424:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574718.149700:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488807552:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574718.149711:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574718.149716:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574718/real 1788574718] req@ffff94d408c56300 x1875450488807552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574718.149725:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488807552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574718.149730:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574718.149731:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574718.149733:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488807552:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574718.149736:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488807680:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574718.149745:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574718.149750:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574718/real 1788574718] req@ffff94d408c55f80 x1875450488807680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574718.149757:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488807680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574718.149762:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574718.149764:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574718.149766:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488807680:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574718.661263:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574719.109347:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574719.685310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574720.133411:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574720.709333:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574721.157437:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574721.733291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574722.181333:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574722.757327:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574723.205414:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574723.269371:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574723.269382:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574723.269387:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574723.269399:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5871 00000100:00080000:0.0:1788574723.269412:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574723.269420:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574723.269464:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488807808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574723.269491:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574723.269495:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574723.269497:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574723.269500:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5871 00000100:00080000:0.0:1788574723.269503:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574723.269506:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574723.269515:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574723.269528:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488807936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574723.269539:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574723.269542:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574723.269544:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574723.269549:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5871 00000100:00080000:0.0:1788574723.269554:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574723.269557:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488807808:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788574723.269558:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574723.269564:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574723.269593:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488808064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574723.269603:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574723.269606:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574723.269608:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574723.269611:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5871 00000100:00080000:0.0:1788574723.269615:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574723.269619:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574723.269624:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574723.269637:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488808192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574723.269650:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574723.269658:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574723.269661:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574723.269669:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5871 00000800:00000100:1.0:1788574723.269675:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788574723.269677:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574723.269684:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574723.269688:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574723.269699:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488808320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574723.269749:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574723/real 1788574723] req@ffff94d408d3ce00 x1875450488807808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574723.269781:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488807808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574723.269792:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574723.269797:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574723.269802:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488807808:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788574723.269831:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488807936:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574723.269859:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574723.269875:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574723/real 1788574723] req@ffff94d408d3d500 x1875450488807936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574723.269889:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488807936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574723.269898:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574723.269902:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574723.269907:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488807936:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574723.269913:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488808064:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574723.269935:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574723.269948:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574723/real 1788574723] req@ffff94d408d3ca80 x1875450488808064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574723.269961:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488808064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574723.269976:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574723.269979:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574723.269982:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488808064:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574723.269989:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488808192:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574723.270009:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574723.270021:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574723/real 1788574723] req@ffff94d408d3ea00 x1875450488808192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574723.270034:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488808192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574723.270042:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574723.270046:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574723.270050:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488808192:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574723.270058:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488808320:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574723.270081:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574723.270094:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574723/real 1788574723] req@ffff94d408d3e680 x1875450488808320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574723.270107:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488808320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574723.270115:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574723.270118:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574723.270122:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488808320:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574723.781317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574724.229397:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574724.805306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574725.253374:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574725.829369:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574726.277430:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574726.853336:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574727.301433:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574727.877311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574728.325367:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574728.389335:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574728.389346:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574728.389351:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574728.389363:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5876 00000100:00080000:1.0:1788574728.389375:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574728.389383:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574728.389429:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488808448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574728.389461:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574728.389464:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574728.389465:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574728.389468:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5876 00000100:00080000:1.0:1788574728.389471:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574728.389473:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574728.389483:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574728.389494:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488808576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574728.389502:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574728.389505:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574728.389507:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574728.389512:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5876 00000100:00080000:1.0:1788574728.389515:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574728.389519:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574728.389527:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574728.389528:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488808448:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788574728.389537:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488808704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574728.389546:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574728.389548:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574728.389551:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574728.389557:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5876 00000100:00080000:1.0:1788574728.389559:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574728.389562:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574728.389566:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574728.389574:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488808832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574728.389581:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574728.389583:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574728.389585:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574728.389589:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5876 00000100:00080000:1.0:1788574728.389591:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574728.389594:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574728.389597:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574728.389600:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574728.389620:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488808960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574728.389644:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574728/real 1788574728] req@ffff94d4049be680 x1875450488808448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574728.389660:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488808448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574728.389669:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574728.389672:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574728.389675:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488808448:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574728.389681:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488808576:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574728.389697:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574728.389704:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574728/real 1788574728] req@ffff94d4049bf100 x1875450488808576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574728.389713:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488808576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574728.389718:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574728.389720:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574728.389722:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488808576:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574728.389742:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488808704:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574728.389752:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574728.389758:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574728/real 1788574728] req@ffff94d4049bca80 x1875450488808704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574728.389766:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488808704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574728.389771:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574728.389773:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574728.389775:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488808704:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574728.389779:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488808832:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574728.389790:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574728.389795:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574728/real 1788574728] req@ffff94d4049bd500 x1875450488808832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574728.389802:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488808832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574728.389812:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574728.389814:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574728.389816:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488808832:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574728.389820:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488808960:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574728.389829:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574728.389834:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574728/real 1788574728] req@ffff94d4049bdc00 x1875450488808960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574728.389841:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488808960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574728.389846:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574728.389848:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574728.389850:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488808960:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574728.901274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574729.349335:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574729.925295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574730.373356:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574730.949340:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574731.397353:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574731.973334:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574732.421398:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574732.997295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574733.445396:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574733.509416:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574733.509427:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574733.509432:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574733.509443:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5881 00000100:00080000:3.0:1788574733.509456:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574733.509464:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574733.509512:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488809088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574733.509541:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574733.509545:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574733.509546:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574733.509549:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5881 00000100:00080000:3.0:1788574733.509558:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574733.509561:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574733.509571:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574733.509579:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488809088:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574733.509582:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488809216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574733.509598:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574733.509601:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574733.509602:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574733.509606:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5881 00000100:00080000:3.0:1788574733.509609:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574733.509611:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574733.509614:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574733.509627:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488809344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574733.509634:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574733.509637:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574733.509638:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574733.509641:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5881 00000100:00080000:3.0:1788574733.509652:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574733.509655:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:1.0:1788574733.509675:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:3.0:1788574733.509679:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574733.509692:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488809472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574733.509700:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574733.509703:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574733.509706:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574733.509710:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5881 00000100:00080000:3.0:1788574733.509715:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574733.509720:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574733.509723:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574733/real 1788574733] req@ffff94d408c57480 x1875450488809088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788574733.509728:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574733.509748:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488809600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574733.509749:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488809088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574733.509783:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574733.509787:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574733.509791:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488809088:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788574733.509816:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488809216:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574733.509836:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574733.509844:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574733/real 1788574733] req@ffff94d408c54000 x1875450488809216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574733.509853:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488809216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574733.509859:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574733.509862:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574733.509864:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488809216:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574733.509869:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488809344:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574733.509881:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574733.509887:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574733/real 1788574733] req@ffff94d408c55f80 x1875450488809344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574733.509896:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488809344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574733.509901:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574733.509903:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574733.509905:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488809344:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574733.509909:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488809472:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574733.509920:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574733.509926:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574733/real 1788574733] req@ffff94d408c56300 x1875450488809472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574733.509933:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488809472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574733.509938:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574733.509940:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574733.509942:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488809472:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574733.509945:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488809600:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574733.509955:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574733.509960:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574733/real 1788574733] req@ffff94d408c54700 x1875450488809600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574733.509968:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488809600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574733.509974:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574733.509975:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574733.509977:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488809600:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574734.021336:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574734.469392:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574735.045280:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574735.493348:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574736.069306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574736.517400:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574737.093256:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574737.541339:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574738.117371:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574738.565347:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574738.629274:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574738.629289:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574738.629294:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574738.629308:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5886 00000100:00080000:3.0:1788574738.629323:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574738.629331:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574738.629377:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488809728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574738.629406:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574738.629411:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574738.629413:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574738.629418:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5886 00000100:00080000:3.0:1788574738.629422:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574738.629427:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574738.629438:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574738.629455:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488809856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574738.629469:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574738.629474:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574738.629476:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788574738.629482:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488809728:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574738.629486:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5886 00000100:00080000:3.0:1788574738.629500:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574738.629507:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574738.629513:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574738.629527:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488809984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574738.629536:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574738.629539:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574738.629540:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574738.629543:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5886 00000100:00080000:3.0:1788574738.629558:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788574738.629564:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574738.629569:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574738.629578:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574738.629594:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488810112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574738.629606:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574738.629610:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574738.629613:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574738.629618:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5886 00000100:00080000:3.0:1788574738.629623:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574738.629626:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574738/real 1788574738] req@ffff94d408c55180 x1875450488809728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574738.629632:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574738.629650:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574738.629673:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488810240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574738.629674:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488809728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574738.629720:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574738.629726:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574738.629730:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488809728:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788574738.629755:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488809856:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574738.629792:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574738.629807:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574738/real 1788574738] req@ffff94d408c55880 x1875450488809856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574738.629827:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488809856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574738.629836:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574738.629839:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574738.629843:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488809856:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574738.629850:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488809984:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574738.629873:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574738.629886:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574738/real 1788574738] req@ffff94d408c54380 x1875450488809984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574738.629897:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488809984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574738.629903:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574738.629905:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574738.629908:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488809984:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574738.629915:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488810112:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574738.629939:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574738.629962:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574738/real 1788574738] req@ffff94d408c57800 x1875450488810112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574738.629973:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488810112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574738.629983:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574738.629986:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574738.629988:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488810112:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574738.629993:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488810240:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574738.630008:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574738.630015:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574738/real 1788574738] req@ffff94d408c56a00 x1875450488810240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574738.630023:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488810240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574738.630028:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574738.630030:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574738.630032:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488810240:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574739.141263:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574739.589429:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574740.165312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574740.613353:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574741.189346:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574741.637423:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574742.213306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574742.661358:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574743.237323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574743.685380:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574743.749315:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574743.749323:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574743.749326:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574743.749334:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5892 00000100:00080000:3.0:1788574743.749343:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574743.749349:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574743.749383:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488810368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574743.749407:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574743.749411:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574743.749412:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574743.749416:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5892 00000100:00080000:3.0:1788574743.749419:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574743.749424:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574743.749433:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574743.749448:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488810496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574743.749454:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488810368:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574743.749458:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574743.749462:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574743.749463:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574743.749467:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5892 00000100:00080000:3.0:1788574743.749470:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574743.749474:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574743.749479:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574743.749487:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488810624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574743.749496:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574743.749499:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574743.749501:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574743.749504:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5892 00000100:00080000:3.0:1788574743.749507:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574743.749511:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:1.0:1788574743.749512:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:3.0:1788574743.749516:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574743.749524:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488810752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574743.749532:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574743.749535:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574743.749536:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574743.749539:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5892 00000100:00080000:3.0:1788574743.749543:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574743.749546:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574743/real 1788574743] req@ffff94d408c57100 x1875450488810368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574743.749546:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574743.749551:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574743.749559:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488810880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574743.749563:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488810368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574743.749580:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574743.749585:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574743.749589:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488810368:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788574743.749612:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488810496:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574743.749633:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574743.749643:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574743/real 1788574743] req@ffff94d408c55c00 x1875450488810496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574743.749651:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488810496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574743.749656:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574743.749659:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574743.749661:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488810496:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574743.749665:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488810624:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574743.749675:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574743.749681:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574743/real 1788574743] req@ffff94d408c54e00 x1875450488810624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574743.749688:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488810624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574743.749694:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574743.749696:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574743.749698:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488810624:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574743.749701:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488810752:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574743.749711:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574743.749716:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574743/real 1788574743] req@ffff94d408c56d80 x1875450488810752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574743.749723:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488810752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574743.749728:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574743.749758:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574743.749760:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488810752:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574743.749764:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488810880:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574743.749779:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574743.749786:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574743/real 1788574743] req@ffff94d408c54a80 x1875450488810880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574743.749794:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488810880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574743.749799:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574743.749801:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574743.749803:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488810880:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574744.261314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574744.709370:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574745.285376:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574745.733331:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574746.309367:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574746.757493:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574747.333284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574747.781439:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574748.357355:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574748.805491:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574748.869427:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574748.869437:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574748.869442:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574748.869455:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5897 00000100:00080000:1.0:1788574748.869467:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574748.869477:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574748.869527:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488811008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574748.869570:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574748.869575:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574748.869578:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574748.869583:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5897 00000100:00080000:1.0:1788574748.869587:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574748.869592:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574748.869606:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574748.869625:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd880 x1875450488811136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574748.869638:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488811008:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788574748.869639:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574748.869645:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574748.869650:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574748.869656:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5897 00000100:00080000:1.0:1788574748.869660:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574748.869663:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574748.869668:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574748.869679:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488811264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574748.869688:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574748.869691:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574748.869692:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574748.869695:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5897 00000100:00080000:1.0:1788574748.869717:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574748.869721:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574748.869724:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574748.869733:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488811392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574748.869739:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574748.869742:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574748.869744:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574748.869747:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5897 00000100:00080000:1.0:1788574748.869750:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574748.869762:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574748.869766:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574748.869777:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488811520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574748.869786:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488811136:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574748.869830:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488811264:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574748.869860:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488811392:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574748.869883:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488811520:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574749.381395:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574749.829430:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574750.405332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574750.853374:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574751.429282:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574751.877377:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788574751.941473:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:2.0:1788574751.941511:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788574751.941524:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788574751.941559:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788574751.941565:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788574751.941570:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788574751.941577:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:0.0:1788574751.941664:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574748/real 1788574751] req@ffff94d4049bca80 x1875450488811520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574751.941718:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488811520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574751.941733:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574751.941740:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574751.941747:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488811520:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574751.941758:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574748/real 1788574751] req@ffff94d4049bd880 x1875450488811136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574751.941772:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd880 x1875450488811136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574751.941781:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574751.941785:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574751.941789:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488811136:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574751.941798:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574748/real 1788574751] req@ffff94d4049bdc00 x1875450488811264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574751.941810:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488811264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574751.941819:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574751.941824:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574751.941827:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488811264:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574751.941836:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574748/real 1788574751] req@ffff94d4049bd500 x1875450488811392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574751.941848:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488811392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574751.941858:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574751.941862:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574751.941866:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488811392:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574751.941875:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574748/real 1788574751] req@ffff94d4049be300 x1875450488811008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574751.941890:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488811008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574751.941899:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574751.941903:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574751.941906:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488811008:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574752.453308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574752.901337:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574753.477272:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574753.925392:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574753.989335:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574753.989343:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574753.989347:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574753.989357:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5902 00000100:00080000:3.0:1788574753.989365:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574753.989372:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574753.989402:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488811648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574753.989426:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574753.989429:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574753.989430:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574753.989433:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5902 00000100:00080000:3.0:1788574753.989435:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574753.989437:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574753.989444:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574753.989453:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488811776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574753.989459:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574753.989461:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574753.989463:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574753.989465:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5902 00000100:00080000:3.0:1788574753.989467:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574753.989469:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574753.989471:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574753.989477:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488811904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574753.989483:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574753.989486:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574753.989487:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574753.989489:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5902 00000100:00080000:3.0:1788574753.989491:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574753.989494:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574753.989499:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574753.989509:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488812032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574753.989520:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788574753.989525:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488811648:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574753.989525:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574753.989530:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574753.989537:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5902 00000100:00080000:3.0:1788574753.989542:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574753.989546:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574753.989551:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574753.989564:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488812160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574753.989608:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574753.989647:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574753/real 1788574753] req@ffff94d408c56680 x1875450488811648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574753.989686:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488811648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574753.989697:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574753.989701:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574753.989706:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488811648:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574753.989714:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488811776:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574753.989741:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574753.989756:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574753/real 1788574753] req@ffff94d408c57b80 x1875450488811776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574753.989775:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488811776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574753.989784:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574753.989787:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574753.989791:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488811776:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574753.989796:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488811904:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574753.989816:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574753.989828:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574753/real 1788574753] req@ffff94d408c54a80 x1875450488811904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574753.989840:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488811904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574753.989848:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574753.989851:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574753.989854:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488811904:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574753.989877:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488812032:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574753.989897:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574753.989907:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574753/real 1788574753] req@ffff94d408c56d80 x1875450488812032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574753.989919:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488812032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574753.989934:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574753.989938:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574753.989941:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488812032:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574753.989948:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488812160:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574753.989968:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574753.989979:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574753/real 1788574753] req@ffff94d408c54e00 x1875450488812160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574753.989991:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488812160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574753.989999:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574753.990003:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574753.990006:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488812160:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574754.501303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574754.949330:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574755.525278:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574755.973377:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574756.549263:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574756.997341:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574757.573799:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574758.021387:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574758.597558:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574759.045426:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574759.109352:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574759.109364:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574759.109369:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574759.109381:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5907 00000100:00080000:3.0:1788574759.109395:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574759.109403:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574759.109450:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488812288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574759.109481:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574759.109484:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574759.109486:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574759.109488:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5907 00000100:00080000:3.0:1788574759.109491:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574759.109493:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574759.109504:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574759.109512:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488812416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574759.109520:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574759.109523:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574759.109524:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574759.109527:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5907 00000100:00080000:3.0:1788574759.109529:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574759.109532:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574759.109535:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574759.109541:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488812544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574759.109548:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574759.109551:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574759.109552:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574759.109558:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5907 00000100:00100000:0.0:1788574759.109560:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488812288:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574759.109561:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574759.109566:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574759.109573:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574759.109579:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488812672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574759.109586:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574759.109588:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574759.109590:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574759.109594:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5907 00000100:00080000:3.0:1788574759.109596:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574759.109599:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574759.109602:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574759.109610:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488812800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574759.109661:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574759.109710:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574759/real 1788574759] req@ffff94d408c55c00 x1875450488812288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574759.109731:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488812288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574759.109745:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574759.109750:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574759.109755:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488812288:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574759.109764:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488812416:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574759.109791:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574759.109804:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574759/real 1788574759] req@ffff94d408c57100 x1875450488812416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574759.109829:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488812416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574759.109839:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574759.109843:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574759.109847:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488812416:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574759.109877:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488812544:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574759.109897:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574759.109910:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574759/real 1788574759] req@ffff94d408c56a00 x1875450488812544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574759.109955:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488812544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574759.109966:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574759.109969:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574759.109973:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488812544:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574759.109980:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488812672:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574759.110004:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574759.110016:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574759/real 1788574759] req@ffff94d408c57800 x1875450488812672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574759.110030:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488812672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574759.110040:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574759.110043:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574759.110047:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488812672:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574759.110054:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488812800:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574759.110073:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574759.110083:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574759/real 1788574759] req@ffff94d408c54380 x1875450488812800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574759.110096:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488812800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574759.110105:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574759.110109:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574759.110113:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488812800:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574759.621564:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574760.069361:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574760.645834:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574761.093430:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574761.669374:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574762.117384:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574762.693328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574763.141380:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574763.717323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574764.165402:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574764.229329:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574764.229337:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574764.229341:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574764.229350:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5912 00000100:00080000:0.0:1788574764.229358:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574764.229364:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574764.229395:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f100 x1875450488812928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574764.229418:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574764.229421:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574764.229423:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574764.229425:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5912 00000100:00080000:0.0:1788574764.229427:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574764.229430:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574764.229436:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574764.229445:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f800 x1875450488813056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574764.229453:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574764.229455:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574764.229456:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574764.229462:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5912 00000100:00100000:1.0:1788574764.229463:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488812928:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788574764.229464:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574764.229471:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574764.229476:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574764.229485:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488813184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574764.229491:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574764.229494:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574764.229495:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574764.229498:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5912 00000100:00080000:0.0:1788574764.229513:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574764.229516:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574764.229520:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788574764.229525:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574764.229530:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488813312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574764.229538:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574764.229540:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574764.229542:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574764.229545:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5912 00000100:00080000:0.0:1788574764.229548:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574764.229550:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574764.229554:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574764.229563:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574764/real 1788574764] req@ffff94d408d3f100 x1875450488812928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574764.229564:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488813440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574764.229609:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f100 x1875450488812928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574764.229623:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574764.229628:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574764.229632:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488812928:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788574764.229653:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488813056:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574764.229673:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574764.229682:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574764/real 1788574764] req@ffff94d408d3f800 x1875450488813056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574764.229691:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f800 x1875450488813056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574764.229697:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574764.229699:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574764.229701:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488813056:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574764.229708:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488813184:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574764.229719:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574764.229725:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574764/real 1788574764] req@ffff94d408d3e680 x1875450488813184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574764.229733:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488813184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574764.229738:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574764.229740:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574764.229742:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488813184:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574764.229745:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488813312:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574764.229756:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574764.229781:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574764/real 1788574764] req@ffff94d408d3ea00 x1875450488813312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574764.229795:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488813312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574764.229801:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574764.229803:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574764.229805:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488813312:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574764.229809:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488813440:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574764.229824:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574764.229832:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574764/real 1788574764] req@ffff94d408d3ca80 x1875450488813440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574764.229840:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488813440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574764.229845:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574764.229847:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574764.229849:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488813440:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574764.741326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574765.189371:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574765.765305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574766.213384:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574766.789309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574767.237352:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574767.813307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574768.261361:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574768.837304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574769.285349:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574769.349334:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574769.349342:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574769.349346:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574769.349354:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5917 00000100:00080000:3.0:1788574769.349363:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574769.349369:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574769.349401:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488813568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574769.349424:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574769.349427:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574769.349428:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574769.349431:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5917 00000100:00080000:3.0:1788574769.349433:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574769.349435:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574769.349442:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574769.349453:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488813696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574769.349460:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574769.349463:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574769.349465:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788574769.349470:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488813568:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574769.349471:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5917 00000100:00080000:3.0:1788574769.349474:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574769.349479:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574769.349483:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574769.349491:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488813824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574769.349497:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574769.349500:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574769.349501:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574769.349503:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5917 00000100:00080000:3.0:1788574769.349506:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574769.349509:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574769.349513:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574769.349521:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488813952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574769.349527:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574769.349530:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574769.349531:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574769.349533:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5917 00000100:00080000:3.0:1788574769.349535:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574769.349537:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574769.349541:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788574769.349545:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574769.349556:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488814080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574769.349585:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574769/real 1788574769] req@ffff94d408c55880 x1875450488813568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574769.349634:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488813568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574769.349647:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574769.349651:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574769.349656:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488813568:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788574769.349677:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488813696:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574769.349697:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574769.349706:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574769/real 1788574769] req@ffff94d408c55180 x1875450488813696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574769.349719:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488813696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574769.349725:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574769.349727:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574769.349729:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488813696:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574769.349733:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488813824:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574769.349744:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574769.349750:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574769/real 1788574769] req@ffff94d408c54700 x1875450488813824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574769.349757:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488813824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574769.349762:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574769.349764:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574769.349766:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488813824:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574769.349770:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488813952:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574769.349779:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574769.349785:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574769/real 1788574769] req@ffff94d408c56300 x1875450488813952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574769.349792:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488813952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574769.349801:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574769.349803:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574769.349805:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488813952:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574769.349808:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488814080:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574769.349817:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574769.349822:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574769/real 1788574769] req@ffff94d408c55f80 x1875450488814080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574769.349828:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488814080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574769.349833:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574769.349835:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574769.349837:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488814080:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574769.861300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574770.309369:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574770.885294:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574771.333325:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574771.909295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574772.357374:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574772.933286:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574773.381491:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574773.957367:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574774.405432:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574774.469334:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574774.469343:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574774.469346:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574774.469354:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5922 00000100:00080000:0.0:1788574774.469364:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574774.469370:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574774.469409:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488814208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574774.469430:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574774.469435:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574774.469436:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574774.469439:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5922 00000100:00080000:0.0:1788574774.469441:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574774.469444:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574774.469462:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574774.469473:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488814336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574774.469483:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574774.469491:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574774.469495:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788574774.469498:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488814208:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788574774.469499:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5922 00000100:00080000:0.0:1788574774.469505:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574774.469513:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574774.469518:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574774.469525:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488814464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574774.469532:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574774.469534:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574774.469535:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574774.469539:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5922 00000100:00080000:0.0:1788574774.469542:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574774.469544:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574774.469548:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574774.469557:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488814592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574774.469563:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574774.469566:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574774.469567:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574774.469570:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5922 00000100:00080000:0.0:1788574774.469591:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574774.469598:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574774.469603:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788574774.469614:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574774.469618:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488814720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574774.469676:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574774/real 1788574774] req@ffff94d408d3d500 x1875450488814208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574774.469699:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488814208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574774.469717:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574774.469721:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574774.469726:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488814208:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788574774.469757:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488814336:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574774.469784:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574774.469797:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574774/real 1788574774] req@ffff94d408d3ce00 x1875450488814336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574774.469809:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488814336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574774.469816:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574774.469819:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574774.469823:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488814336:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574774.469829:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488814464:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788574774.469848:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574774.469859:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574774/real 1788574774] req@ffff94d408d3df80 x1875450488814464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574774.469871:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488814464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574774.469879:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574774.469882:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574774.469886:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488814464:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574774.469893:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488814592:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574774.469913:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574774.469966:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574774/real 1788574774] req@ffff94d408d3d180 x1875450488814592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574774.469978:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488814592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574774.469986:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574774.469990:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574774.469993:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488814592:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574774.469999:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488814720:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788574774.470020:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574774.470030:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574774/real 1788574774] req@ffff94d408d3c000 x1875450488814720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574774.470041:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488814720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574774.470048:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574774.470052:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574774.470055:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488814720:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574774.981357:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574775.429455:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574776.005281:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574776.453345:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574777.029307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574777.477397:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574778.053271:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574778.501405:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574779.077352:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574779.525431:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574779.589362:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574779.589373:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574779.589378:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574779.589389:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5927 00000100:00080000:1.0:1788574779.589400:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574779.589407:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574779.589465:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488814848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574779.589497:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574779.589501:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574779.589504:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574779.589508:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5927 00000100:00080000:1.0:1788574779.589513:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574779.589518:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574779.589529:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574779.589534:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488814848:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788574779.589543:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488814976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574779.589552:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574779.589554:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574779.589556:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574779.589559:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5927 00000100:00080000:1.0:1788574779.589562:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574779.589565:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574779.589568:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574779.589576:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488815104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574779.589583:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574779.589585:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574779.589586:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574779.589589:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5927 00000100:00080000:1.0:1788574779.589600:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574779.589603:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574779.589607:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574779.589614:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488815232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574779.589622:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788574779.589622:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574779.589628:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574779.589629:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574779.589632:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5927 00000100:00080000:1.0:1788574779.589635:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574779.589637:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574779.589639:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574779.589644:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488815360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574779.589682:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574779/real 1788574779] req@ffff94d4049bf100 x1875450488814848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574779.589702:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488814848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574779.589714:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574779.589718:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574779.589722:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488814848:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574779.589748:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488814976:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574779.589770:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574779.589781:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574779/real 1788574779] req@ffff94d4049be680 x1875450488814976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574779.589793:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488814976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574779.589800:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574779.589803:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574779.589806:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488814976:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574779.589812:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488815104:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574779.589828:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574779.589836:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574779/real 1788574779] req@ffff94d4049bf800 x1875450488815104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574779.589846:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488815104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574779.589854:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574779.589857:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574779.589860:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488815104:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574779.589866:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488815232:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574779.589894:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574779.589904:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574779/real 1788574779] req@ffff94d4049bed80 x1875450488815232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574779.589921:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488815232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574779.589929:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574779.589932:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574779.589935:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488815232:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574779.589941:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488815360:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574779.589957:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574779.589965:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574779/real 1788574779] req@ffff94d4049bc700 x1875450488815360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574779.589975:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488815360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574779.589983:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574779.589986:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574779.589989:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488815360:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574780.101404:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574780.549379:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574781.125335:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574781.573354:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574782.149347:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574782.597361:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574783.173324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574783.621345:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574784.197357:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574784.645328:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574784.709299:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574784.709307:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574784.709311:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574784.709321:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5933 00000100:00080000:3.0:1788574784.709329:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574784.709335:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574784.709368:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488815488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574784.709394:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574784.709398:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574784.709400:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574784.709405:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5933 00000100:00080000:3.0:1788574784.709410:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574784.709415:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574784.709423:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488815488:192.168.202.125@tcp:250:kworker.0 00000008:00100000:3.0:1788574784.709425:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574784.709443:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488815616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574784.709454:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574784.709458:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574784.709460:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574784.709463:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5933 00000800:00000100:0.0:1788574784.709482:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574784.709520:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574784.709525:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574784.709530:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574784.709540:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488815744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574784.709547:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574784/real 1788574784] req@ffff94d408c54000 x1875450488815488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574784.709548:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574784.709555:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574784.709558:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574784.709563:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5933 00000100:00100000:0.0:1788574784.709571:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488815488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574784.709582:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574784.709585:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574784.709588:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488815488:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574784.709605:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488815616:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574784.709621:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574784.709629:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574784/real 1788574784] req@ffff94d408c57480 x1875450488815616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574784.709641:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488815616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574784.709649:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574784.709652:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574784.709656:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488815616:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574784.709663:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488815744:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788574784.709669:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574784.709674:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574784.709679:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788574784.709684:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574784.709691:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488815872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574784.709699:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788574784.709702:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574784/real 1788574784] req@ffff94d408c55f80 x1875450488815744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574784.709703:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1788574784.709718:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488815744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574784.709727:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574784.709730:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574784.709732:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488815744:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574784.709751:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488815872:192.168.202.125@tcp:8:kworker.0 00000100:00080000:3.0:1788574784.709773:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788574784.709774:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574784.709782:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5933 00000100:00080000:3.0:1788574784.709787:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788574784.709792:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574784/real 1788574784] req@ffff94d408c56300 x1875450488815872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574784.709795:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574784.709804:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574784.709811:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488815872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574784.709819:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488816000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574784.709822:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574784.709830:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574784.709836:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488815872:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574784.709851:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488816000:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574784.709869:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574784.709879:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574784/real 1788574784] req@ffff94d408c54700 x1875450488816000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574784.709890:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488816000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574784.709898:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574784.709901:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574784.709904:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488816000:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574785.221320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574785.669377:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574786.245326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574786.693373:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574787.269345:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574787.717367:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574788.293358:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574788.741374:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574789.317372:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574789.765359:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574789.829330:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574789.829341:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574789.829345:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574789.829356:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5938 00000100:00080000:3.0:1788574789.829369:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574789.829377:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574789.829430:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488816128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574789.829458:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574789.829461:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574789.829463:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574789.829467:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5938 00000100:00080000:3.0:1788574789.829471:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574789.829475:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574789.829486:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574789.829500:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488816256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574789.829510:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574789.829516:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788574789.829521:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488816128:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574789.829521:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574789.829536:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5938 00000100:00080000:3.0:1788574789.829540:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574789.829543:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574789.829547:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574789.829559:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488816384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574789.829566:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574789.829569:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574789.829571:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574789.829574:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5938 00000100:00080000:3.0:1788574789.829576:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574789.829579:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574789.829583:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788574789.829591:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574789.829592:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488816512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574789.829602:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574789.829605:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574789.829606:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574789.829610:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5938 00000100:00080000:3.0:1788574789.829616:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574789.829620:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574789.829625:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574789.829633:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574789/real 1788574789] req@ffff94d408c55180 x1875450488816128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574789.829634:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488816640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574789.829700:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488816128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574789.829709:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574789.829713:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574789.829718:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488816128:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574789.829742:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488816256:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574789.829765:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574789.829778:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574789/real 1788574789] req@ffff94d408c55880 x1875450488816256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574789.829787:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488816256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574789.829792:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574789.829794:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574789.829796:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488816256:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574789.829800:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488816384:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574789.829821:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574789.829829:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574789/real 1788574789] req@ffff94d408c54380 x1875450488816384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574789.829836:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488816384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574789.829844:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574789.829846:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574789.829848:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488816384:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574789.829851:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488816512:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574789.829862:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574789.829867:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574789/real 1788574789] req@ffff94d408c57800 x1875450488816512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574789.829873:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488816512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574789.829877:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574789.829879:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574789.829880:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488816512:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574789.829883:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488816640:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574789.829892:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574789.829897:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574789/real 1788574789] req@ffff94d408c56a00 x1875450488816640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574789.829903:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488816640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574789.829907:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574789.829908:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574789.829911:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488816640:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574790.341281:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574790.789316:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574791.365272:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574791.813350:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574792.389280:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574792.837326:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574793.413347:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574793.861369:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574794.437360:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574794.885368:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574794.949342:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574794.949356:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574794.949361:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574794.949375:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5943 00000100:00080000:3.0:1788574794.949391:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574794.949399:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574794.949449:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488816768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574794.949485:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574794.949490:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574794.949492:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574794.949496:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5943 00000100:00080000:3.0:1788574794.949500:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574794.949504:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574794.949516:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574794.949531:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488816896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574794.949542:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574794.949547:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574794.949549:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574794.949553:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5943 00000100:00080000:3.0:1788574794.949556:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788574794.949562:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488816768:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574794.949563:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574794.949572:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574794.949584:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488817024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574794.949595:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574794.949599:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574794.949601:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574794.949605:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5943 00000100:00080000:3.0:1788574794.949609:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574794.949613:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574794.949618:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574794.949629:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488817152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574794.949639:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574794.949643:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574794.949646:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574794.949650:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5943 00000100:00080000:3.0:1788574794.949654:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788574794.949659:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574794.949663:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574794.949669:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574794.949681:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488817280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574794.949712:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574794/real 1788574794] req@ffff94d408c57100 x1875450488816768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574794.949736:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488816768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574794.949749:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574794.949753:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574794.949757:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488816768:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574794.949764:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488816896:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574794.949789:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574794.949804:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574794/real 1788574794] req@ffff94d408c55c00 x1875450488816896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574794.949818:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488816896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574794.949862:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574794.949866:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574794.949870:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488816896:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574794.949900:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488817024:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574794.949925:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574794.949940:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574794/real 1788574794] req@ffff94d408c54e00 x1875450488817024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574794.949953:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488817024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574794.949963:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574794.949966:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574794.949970:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488817024:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574794.949977:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488817152:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574794.950010:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574794.950023:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574794/real 1788574794] req@ffff94d408c56d80 x1875450488817152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574794.950037:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488817152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574794.950046:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574794.950050:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574794.950053:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488817152:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574794.950060:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488817280:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574794.950081:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574794.950092:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574794/real 1788574794] req@ffff94d408c54a80 x1875450488817280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574794.950101:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488817280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574794.950111:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574794.950115:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574794.950118:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488817280:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574795.461282:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574795.909366:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574796.485271:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574796.933449:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574797.509310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574797.957429:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574798.533396:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574798.981464:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574799.557287:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574800.005414:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574800.069364:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574800.069376:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574800.069380:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574800.069392:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5948 00000100:00080000:1.0:1788574800.069405:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574800.069413:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574800.069462:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488817408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574800.069489:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574800.069492:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574800.069494:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574800.069497:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5948 00000100:00080000:1.0:1788574800.069500:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574800.069502:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574800.069512:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574800.069522:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488817536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574800.069531:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574800.069535:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574800.069536:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574800.069542:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488817408:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788574800.069543:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5948 00000100:00080000:1.0:1788574800.069548:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574800.069553:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574800.069556:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574800.069564:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488817664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574800.069571:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574800.069574:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574800.069575:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574800.069578:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5948 00000100:00080000:1.0:1788574800.069579:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574800.069582:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574800.069585:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574800.069592:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488817792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574800.069599:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574800.069601:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574800.069602:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574800.069604:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5948 00000100:00080000:1.0:1788574800.069608:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574800.069611:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574800.069615:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574800.069622:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488817920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574800.069631:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574800.069722:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574800/real 1788574800] req@ffff94d4049bc000 x1875450488817408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574800.069742:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488817408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574800.069751:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574800.069754:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574800.069758:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488817408:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574800.069782:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488817536:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574800.069802:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574800.069812:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574800/real 1788574800] req@ffff94d4049bfb80 x1875450488817536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574800.069820:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488817536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574800.069826:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574800.069829:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574800.069831:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488817536:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574800.069835:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488817664:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574800.069848:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574800.069859:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574800/real 1788574800] req@ffff94d4049bd180 x1875450488817664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574800.069872:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488817664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574800.069881:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574800.069885:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574800.069889:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488817664:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574800.069896:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488817792:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574800.069919:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574800.069930:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574800/real 1788574800] req@ffff94d4049bce00 x1875450488817792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574800.069938:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488817792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574800.069943:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574800.069946:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574800.069948:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488817792:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574800.069951:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488817920:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574800.069964:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574800.069970:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574800/real 1788574800] req@ffff94d4049bea00 x1875450488817920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574800.069978:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488817920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574800.069983:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574800.069984:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574800.069986:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488817920:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574800.581339:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574801.029460:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574801.605312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574802.053373:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574802.629369:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574803.077421:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574803.653266:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574804.101389:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574804.677280:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574805.125429:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574805.189380:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574805.189391:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574805.189396:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574805.189409:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5953 00000100:00080000:1.0:1788574805.189422:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574805.189431:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574805.189485:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488818048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574805.189528:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574805.189533:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574805.189535:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574805.189539:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5953 00000100:00080000:1.0:1788574805.189542:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574805.189547:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574805.189557:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574805.189566:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488818048:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788574805.189571:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488818176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574805.189583:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574805.189587:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574805.189589:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574805.189592:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5953 00000100:00080000:1.0:1788574805.189594:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574805.189596:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574805.189600:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574805.189611:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488818304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574805.189617:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574805.189620:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574805.189621:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574805.189624:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5953 00000100:00080000:1.0:1788574805.189641:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574805.189645:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574805.189650:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574805.189664:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488818432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574805.189673:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788574805.189685:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574805.189690:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574805.189692:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574805.189698:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5953 00000100:00080000:1.0:1788574805.189701:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574805.189705:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574805.189711:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574805.189721:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488818560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574805.189727:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574805/real 1788574805] req@ffff94d4049bf480 x1875450488818048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574805.189749:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488818048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574805.189758:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574805.189762:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574805.189765:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488818048:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574805.189792:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488818176:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574805.189811:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574805.189820:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574805/real 1788574805] req@ffff94d4049bc380 x1875450488818176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574805.189832:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc380 x1875450488818176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574805.189847:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574805.189852:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574805.189856:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488818176:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574805.189872:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488818304:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574805.189893:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574805.189902:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574805/real 1788574805] req@ffff94d4049bea00 x1875450488818304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574805.189911:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488818304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574805.189917:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574805.189920:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574805.189922:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488818304:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574805.189927:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488818432:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574805.189940:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574805.189946:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574805/real 1788574805] req@ffff94d4049bce00 x1875450488818432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574805.189954:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488818432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574805.189967:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574805.189969:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574805.189972:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488818432:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574805.189976:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488818560:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574805.189989:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574805.189995:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574805/real 1788574805] req@ffff94d4049bd180 x1875450488818560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574805.190003:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488818560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574805.190008:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574805.190009:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574805.190011:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488818560:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574805.701279:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574806.149461:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574806.725287:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574807.173436:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574807.749263:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574808.197361:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574808.773273:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574809.221465:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574809.797346:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574810.245339:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574810.309318:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574810.309328:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574810.309332:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574810.309343:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5958 00000100:00080000:1.0:1788574810.309352:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574810.309374:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574810.309410:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488818688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574810.309436:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574810.309440:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574810.309441:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574810.309445:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5958 00000100:00080000:1.0:1788574810.309448:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574810.309451:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574810.309460:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574810.309474:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488818816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574810.309483:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788574810.309502:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488818688:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788574810.309502:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574810.309505:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574810.309514:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5958 00000100:00080000:1.0:1788574810.309518:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574810.309521:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574810.309526:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574810.309538:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488818944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574810.309554:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574810.309558:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574810.309560:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574810.309563:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5958 00000100:00080000:1.0:1788574810.309567:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788574810.309569:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788574810.309575:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574810.309582:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574810.309593:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488819072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574810.309601:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574810.309604:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574810.309606:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574810.309610:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574810/real 1788574810] req@ffff94d4049bfb80 x1875450488818688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574810.309612:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5958 00000100:00080000:1.0:1788574810.309618:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574810.309625:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574810.309632:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574810.309640:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488818688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574810.309646:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488819200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574810.309661:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574810.309667:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574810.309672:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488818688:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574810.309697:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488818816:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574810.309719:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574810.309731:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574810/real 1788574810] req@ffff94d4049bc000 x1875450488818816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574810.309744:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488818816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574810.309752:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574810.309755:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574810.309759:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488818816:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574810.309764:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488818944:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574810.309782:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574810.309792:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574810/real 1788574810] req@ffff94d4049bc700 x1875450488818944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574810.309803:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488818944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574810.309810:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574810.309814:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574810.309817:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488818944:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574810.309823:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488819072:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574810.309840:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574810.309850:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574810/real 1788574810] req@ffff94d4049bed80 x1875450488819072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574810.309861:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488819072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574810.309869:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574810.309872:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574810.309875:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488819072:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574810.309881:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488819200:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574810.309897:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574810.309906:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574810/real 1788574810] req@ffff94d4049bf800 x1875450488819200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574810.309917:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488819200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574810.309924:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574810.309926:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574810.309929:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488819200:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574810.821334:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574811.269411:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574811.845328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574812.293392:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574812.869384:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574813.317415:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574813.893392:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574814.341467:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574814.917349:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574815.365350:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574815.429364:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574815.429379:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574815.429385:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574815.429398:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5963 00000100:00080000:0.0:1788574815.429411:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574815.429419:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574815.429463:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488819328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574815.429491:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574815.429494:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574815.429495:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574815.429498:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5963 00000100:00080000:0.0:1788574815.429501:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574815.429503:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574815.429512:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574815.429521:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488819456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574815.429529:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574815.429532:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574815.429534:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788574815.429534:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488819328:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788574815.429539:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5963 00000100:00080000:0.0:1788574815.429545:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574815.429547:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574815.429550:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574815.429556:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488819584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574815.429562:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574815.429564:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574815.429566:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574815.429568:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5963 00000100:00080000:0.0:1788574815.429569:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574815.429572:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574815.429576:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574815.429584:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488819712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574815.429590:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574815.429592:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574815.429593:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574815.429595:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5963 00000100:00080000:0.0:1788574815.429605:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574815.429608:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574815.429610:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574815.429617:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488819840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574815.429672:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488819456:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574815.429707:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488819584:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574815.429735:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488819712:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574815.429760:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488819840:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574815.941344:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574816.389437:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574816.965286:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574817.413449:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574817.989362:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574818.437411:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788574818.501557:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:2.0:1788574818.501615:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788574818.501636:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788574818.501684:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788574818.501693:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788574818.501702:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788574818.501711:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:1.0:1788574818.501755:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574815/real 1788574818] req@ffff94d408d3f480 x1875450488819456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574818.501790:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488819456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574818.501817:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574818.501823:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574818.501829:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488819456:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574818.501838:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574815/real 1788574818] req@ffff94d408d3c700 x1875450488819584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574818.501858:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488819584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574818.501867:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574818.501871:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574818.501874:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488819584:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574818.501881:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574815/real 1788574818] req@ffff94d408d3e300 x1875450488819712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574818.501892:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488819712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574818.501900:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574818.501904:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574818.501906:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488819712:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574818.501913:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574815/real 1788574818] req@ffff94d408d3dc00 x1875450488819840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574818.501924:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488819840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574818.501930:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574818.501943:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574818.501950:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488819840:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788574818.501957:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574815/real 1788574818] req@ffff94d408d3c380 x1875450488819328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574818.501969:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488819328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574818.501976:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574818.501979:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788574818.501982:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488819328:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574819.013295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574819.461413:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574820.037362:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574820.485348:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574820.549302:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574820.549311:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574820.549315:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574820.549323:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5968 00000100:00080000:1.0:1788574820.549332:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574820.549340:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574820.549372:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488819968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574820.549397:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574820.549399:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574820.549401:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574820.549404:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5968 00000100:00080000:1.0:1788574820.549413:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574820.549418:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574820.549426:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574820.549436:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488820096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574820.549445:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574820.549451:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788574820.549451:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488819968:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788574820.549461:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574820.549469:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5968 00000100:00080000:1.0:1788574820.549474:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574820.549479:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574820.549486:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574820.549499:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488820224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574820.549510:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574820.549514:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574820.549516:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574820.549521:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5968 00000100:00080000:1.0:1788574820.549526:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574820.549531:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574820.549537:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574820.549549:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488820352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574820.549563:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788574820.549565:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574820.549570:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574820.549580:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574820.549587:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5968 00000100:00080000:1.0:1788574820.549593:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574820.549598:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574820.549603:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574820.549620:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574820/real 1788574820] req@ffff94d4049be680 x1875450488819968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574820.549622:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488820480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574820.549724:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488819968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574820.549741:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574820.549746:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574820.549750:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488819968:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574820.549783:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488820096:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574820.549812:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574820.549826:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574820/real 1788574820] req@ffff94d4049bf100 x1875450488820096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574820.549839:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488820096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574820.549848:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574820.549851:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574820.549854:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488820096:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574820.549862:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488820224:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574820.549882:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574820.549893:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574820/real 1788574820] req@ffff94d4049be300 x1875450488820224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574820.549903:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488820224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574820.549911:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574820.549914:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574820.549917:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488820224:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574820.549924:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488820352:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574820.549945:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574820.549955:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574820/real 1788574820] req@ffff94d4049bd500 x1875450488820352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574820.549965:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488820352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574820.549973:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574820.549977:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574820.549980:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488820352:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574820.549986:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488820480:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574820.550006:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574820.550015:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574820/real 1788574820] req@ffff94d4049bdc00 x1875450488820480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574820.550026:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488820480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574820.550033:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574820.550036:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574820.550039:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488820480:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574821.061310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574821.509335:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574822.085293:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574822.533361:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574823.109332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574823.557383:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574824.133408:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574824.581362:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574825.157576:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574825.605361:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574825.669346:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574825.669360:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574825.669365:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574825.669380:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5973 00000100:00080000:1.0:1788574825.669395:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574825.669405:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574825.669468:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd880 x1875450488820608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574825.669502:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574825.669506:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574825.669508:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574825.669512:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5973 00000100:00080000:1.0:1788574825.669516:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574825.669522:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574825.669534:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574825.669548:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488820736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574825.669566:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574825.669566:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488820608:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788574825.669579:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574825.669583:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574825.669588:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5973 00000100:00080000:1.0:1788574825.669593:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574825.669599:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574825.669605:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574825.669626:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488820864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574825.669636:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574825.669641:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574825.669643:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574825.669647:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5973 00000100:00080000:1.0:1788574825.669652:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788574825.669652:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788574825.669662:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574825.669669:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574825.669687:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488820992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574825.669694:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574825/real 1788574825] req@ffff94d4049bd880 x1875450488820608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574825.669727:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574825.669733:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574825.669737:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574825.669738:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd880 x1875450488820608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574825.669755:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5973 00000100:00080000:2.0:1788574825.669763:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574825.669764:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574825.669773:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:1.0:1788574825.669779:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574825.669783:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488820608:192.168.202.125@tcp:250:kworker.0 00000008:00100000:1.0:1788574825.669795:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574825.669815:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488821120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574825.669817:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488820736:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574825.669863:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574825.669877:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574825/real 1788574825] req@ffff94d4049bca80 x1875450488820736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574825.669887:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488820736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574825.669893:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574825.669896:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574825.669898:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488820736:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574825.669903:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488820864:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574825.669915:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574825.669921:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574825/real 1788574825] req@ffff94d4049bdc00 x1875450488820864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574825.669929:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488820864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574825.669939:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574825.669941:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574825.669943:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488820864:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574825.669947:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488820992:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574825.669993:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574825.670001:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574825/real 1788574825] req@ffff94d4049bd500 x1875450488820992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574825.670009:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488820992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574825.670014:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574825.670016:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574825.670019:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488820992:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574825.670038:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488821120:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574825.670051:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574825.670058:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574825/real 1788574825] req@ffff94d4049be300 x1875450488821120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574825.670066:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488821120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574825.670071:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574825.670073:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574825.670076:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488821120:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574826.181700:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574826.629368:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574827.205344:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574827.653387:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574828.229649:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574828.677365:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574829.253363:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574829.701330:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574830.277301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574830.725381:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574830.789339:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574830.789346:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574830.789350:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574830.789358:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5979 00000100:00080000:3.0:1788574830.789368:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574830.789374:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574830.789414:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488821248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574830.789442:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574830.789445:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574830.789447:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574830.789450:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5979 00000100:00080000:3.0:1788574830.789453:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574830.789456:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574830.789463:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574830.789474:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488821376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574830.789482:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574830.789485:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574830.789486:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574830.789491:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5979 00000100:00100000:2.0:1788574830.789493:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488821248:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574830.789494:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574830.789499:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574830.789504:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574830.789515:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488821504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574830.789522:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574830.789524:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574830.789525:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574830.789528:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5979 00000100:00080000:3.0:1788574830.789530:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574830.789533:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574830.789536:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574830.789543:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488821632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574830.789549:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574830.789552:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574830.789553:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574830.789555:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5979 00000100:00080000:3.0:1788574830.789557:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574830.789560:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:2.0:1788574830.789564:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:3.0:1788574830.789567:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574830.789576:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488821760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574830.789598:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574830/real 1788574830] req@ffff94d408c57b80 x1875450488821248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574830.789613:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488821248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574830.789620:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574830.789623:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574830.789626:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488821248:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574830.789644:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488821376:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574830.789660:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574830.789668:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574830/real 1788574830] req@ffff94d408c56680 x1875450488821376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574830.789675:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488821376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574830.789681:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574830.789683:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574830.789686:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488821376:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574830.789690:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488821504:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574830.789700:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574830.789706:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574830/real 1788574830] req@ffff94d408c54a80 x1875450488821504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574830.789713:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488821504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574830.789718:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574830.789720:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574830.789722:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488821504:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574830.789725:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488821632:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574830.789735:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574830.789741:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574830/real 1788574830] req@ffff94d408c56d80 x1875450488821632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574830.789748:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488821632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574830.789752:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574830.789754:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574830.789756:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488821632:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574830.789759:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488821760:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574830.789768:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574830.789774:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574830/real 1788574830] req@ffff94d408c54e00 x1875450488821760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574830.789781:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488821760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574830.789786:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574830.789788:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574830.789790:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488821760:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574831.301271:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574831.749380:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574832.325306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574832.773366:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574833.349311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574833.797372:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574834.373284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574834.821360:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574835.397310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574835.845414:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574835.909332:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574835.909343:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574835.909348:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574835.909358:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5984 00000100:00080000:3.0:1788574835.909378:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574835.909386:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574835.909431:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488821888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574835.909458:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574835.909461:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574835.909463:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574835.909465:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5984 00000100:00080000:3.0:1788574835.909468:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574835.909476:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574835.909488:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574835.909498:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488822016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574835.909506:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574835.909509:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574835.909511:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574835.909515:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5984 00000100:00080000:3.0:1788574835.909517:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574835.909519:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574835.909521:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574835.909527:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488822144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574835.909535:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574835.909537:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574835.909538:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574835.909543:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5984 00000100:00080000:3.0:1788574835.909546:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788574835.909555:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488821888:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574835.909555:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574835.909564:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574835.909577:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488822272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574835.909587:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574835.909589:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574835.909590:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574835.909593:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5984 00000100:00080000:3.0:1788574835.909595:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574835.909597:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574835.909599:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574835.909606:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488822400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574835.909661:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574835.909721:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574835/real 1788574835] req@ffff94d408c55c00 x1875450488821888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574835.909750:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488821888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574835.909763:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574835.909767:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574835.909772:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488821888:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574835.909781:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488822016:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574835.909808:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574835.909822:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574835/real 1788574835] req@ffff94d408c57100 x1875450488822016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574835.909836:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488822016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574835.909845:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574835.909849:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574835.909852:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488822016:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574835.909880:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488822144:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574835.909904:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574835.909916:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574835/real 1788574835] req@ffff94d408c56a00 x1875450488822144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574835.909931:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488822144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574835.909941:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574835.909945:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574835.909948:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488822144:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574835.909955:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488822272:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574835.909978:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574835.909990:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574835/real 1788574835] req@ffff94d408c57800 x1875450488822272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574835.910003:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488822272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574835.910012:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574835.910016:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574835.910019:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488822272:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574835.910026:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488822400:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574835.910045:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574835.910068:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574835/real 1788574835] req@ffff94d408c54380 x1875450488822400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574835.910081:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488822400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574835.910090:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574835.910093:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574835.910097:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488822400:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574836.421264:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574836.869338:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574837.445345:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574837.893386:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574838.469298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574838.917361:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574839.493273:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574839.941379:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574840.517320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574840.965402:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574841.029311:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574841.029322:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574841.029326:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574841.029335:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5989 00000100:00080000:3.0:1788574841.029345:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574841.029351:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574841.029383:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488822528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574841.029409:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574841.029412:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574841.029413:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574841.029416:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5989 00000100:00080000:3.0:1788574841.029418:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574841.029421:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574841.029428:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574841.029438:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488822656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574841.029446:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574841.029448:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574841.029450:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574841.029452:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5989 00000100:00080000:3.0:1788574841.029455:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788574841.029457:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488822528:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574841.029461:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574841.029467:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574841.029476:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488822784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574841.029482:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574841.029485:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574841.029487:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574841.029490:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5989 00000100:00080000:3.0:1788574841.029505:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574841.029508:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574841.029512:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574841.029514:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574841.029523:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488822912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574841.029532:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574841.029534:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574841.029535:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574841.029538:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5989 00000100:00080000:3.0:1788574841.029541:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574841.029544:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574841.029547:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574841.029550:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574841/real 1788574841] req@ffff94d408c55880 x1875450488822528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574841.029557:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488823040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574841.029604:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488822528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574841.029617:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574841.029622:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574841.029626:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488822528:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574841.029632:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488822656:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574841.029652:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574841.029661:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574841/real 1788574841] req@ffff94d408c55180 x1875450488822656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574841.029670:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488822656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574841.029675:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574841.029678:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574841.029680:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488822656:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574841.029698:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488822784:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574841.029709:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574841.029715:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574841/real 1788574841] req@ffff94d408c54700 x1875450488822784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574841.029723:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488822784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574841.029728:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574841.029730:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574841.029732:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488822784:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574841.029736:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488822912:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574841.029746:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574841.029751:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574841/real 1788574841] req@ffff94d408c56300 x1875450488822912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574841.029759:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488822912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574841.029763:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574841.029765:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574841.029767:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488822912:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574841.029770:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488823040:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574841.029779:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574841.029787:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574841/real 1788574841] req@ffff94d408c55f80 x1875450488823040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574841.029799:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488823040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574841.029807:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574841.029811:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574841.029815:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488823040:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574841.541319:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574841.989356:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574842.565308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574843.013382:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574843.589332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574844.037422:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574844.613306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574845.061405:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574845.637338:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574846.085376:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574846.149359:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574846.149371:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574846.149375:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574846.149386:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5994 00000100:00080000:1.0:1788574846.149398:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574846.149408:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574846.149477:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488823168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574846.149511:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574846.149516:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574846.149518:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574846.149522:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5994 00000100:00080000:1.0:1788574846.149526:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574846.149532:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574846.149542:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574846.149555:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488823168:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788574846.149562:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488823296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574846.149577:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574846.149581:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574846.149584:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574846.149588:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5994 00000100:00080000:1.0:1788574846.149593:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788574846.149609:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788574846.149617:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574846.149624:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574846.149641:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574846/real 1788574846] req@ffff94d4049bf100 x1875450488823168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574846.149643:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488823424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574846.149678:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574846.149684:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1788574846.149685:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488823168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574846.149688:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574846.149699:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5994 00000100:00080000:2.0:1788574846.149702:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574846.149706:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574846.149707:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574846.149713:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488823168:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788574846.149717:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574846.149726:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574846.149735:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488823296:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574846.149743:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488823552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574846.149754:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788574846.149760:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574846.149766:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1788574846.149767:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574846/real 1788574846] req@ffff94d4049be680 x1875450488823296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574846.149770:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574846.149781:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5994 00000100:00100000:2.0:1788574846.149786:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488823296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574846.149789:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574846.149797:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574846.149798:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00080000:2.0:1788574846.149803:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574846.149807:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488823296:192.168.202.125@tcp:38:kworker.0 00000008:00100000:1.0:1788574846.149809:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574846.149813:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488823424:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574846.149827:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488823680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574846.149839:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574846.149850:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574846/real 1788574846] req@ffff94d4049bf800 x1875450488823424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574846.149860:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488823424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574846.149870:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574846.149873:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574846.149875:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488823424:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574846.149882:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488823552:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574846.149894:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574846.149900:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574846/real 1788574846] req@ffff94d4049bed80 x1875450488823552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574846.149907:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488823552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574846.149912:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574846.149914:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574846.149916:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488823552:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574846.149919:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488823680:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574846.149928:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574846.149934:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574846/real 1788574846] req@ffff94d4049bc700 x1875450488823680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574846.149941:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488823680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574846.149945:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574846.149947:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574846.149949:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488823680:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574846.661314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574847.109341:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574847.685262:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574848.133373:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574848.709283:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574849.157345:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574849.733329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574850.181403:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574850.757269:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574851.205352:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574851.269332:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574851.269342:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574851.269347:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574851.269356:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 5999 00000100:00080000:1.0:1788574851.269372:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574851.269380:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574851.269418:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488823808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574851.269447:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574851.269459:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574851.269461:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574851.269466:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5999 00000100:00080000:1.0:1788574851.269471:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574851.269476:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574851.269486:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574851.269496:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488823808:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788574851.269499:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488823936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574851.269513:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574851.269516:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574851.269517:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574851.269521:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5999 00000100:00080000:1.0:1788574851.269523:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574851.269526:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574851.269529:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574851.269538:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488824064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574851.269544:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574851.269547:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574851.269548:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574851.269550:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5999 00000100:00080000:1.0:1788574851.269552:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574851.269555:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574851.269560:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574851.269567:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488824192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574851.269573:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788574851.269579:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574851.269582:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574851.269584:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574851.269587:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 5999 00000100:00080000:1.0:1788574851.269589:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574851.269591:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574851.269593:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574851.269599:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488824320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574851.269626:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574851/real 1788574851] req@ffff94d4049bc000 x1875450488823808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574851.269645:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488823808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574851.269656:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574851.269660:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574851.269664:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488823808:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574851.269688:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488823936:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574851.269711:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574851.269721:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574851/real 1788574851] req@ffff94d4049bfb80 x1875450488823936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574851.269729:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488823936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574851.269736:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574851.269738:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574851.269740:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488823936:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574851.269744:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488824064:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574851.269755:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574851.269761:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574851/real 1788574851] req@ffff94d4049bd180 x1875450488824064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574851.269769:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488824064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574851.269774:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574851.269775:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574851.269777:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488824064:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574851.269781:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488824192:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574851.269790:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574851.269796:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574851/real 1788574851] req@ffff94d4049bce00 x1875450488824192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574851.269803:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488824192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574851.269808:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574851.269810:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574851.269811:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488824192:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574851.269815:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488824320:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574851.269824:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574851.269829:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574851/real 1788574851] req@ffff94d4049bea00 x1875450488824320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574851.269835:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488824320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574851.269841:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574851.269843:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574851.269845:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488824320:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574851.781315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574852.229408:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574852.805262:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574853.253398:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574853.829337:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574854.277382:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574854.853327:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574855.301374:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574855.877281:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574856.325341:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574856.389321:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574856.389329:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574856.389333:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574856.389343:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6004 00000100:00080000:1.0:1788574856.389352:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574856.389358:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574856.389390:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488824448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574856.389410:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574856.389413:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574856.389415:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574856.389419:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6004 00000100:00080000:1.0:1788574856.389422:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574856.389425:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574856.389433:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574856.389443:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488824576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574856.389450:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574856.389453:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574856.389454:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574856.389456:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6004 00000100:00080000:1.0:1788574856.389459:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574856.389461:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574856.389463:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574856.389471:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488824704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574856.389477:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574856.389479:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574856.389480:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574856.389492:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6004 00000100:00080000:1.0:1788574856.389495:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574856.389499:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574856.389503:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574856.389513:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488824832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574856.389521:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574856.389524:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574856.389525:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574856.389528:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6004 00000100:00080000:1.0:1788574856.389531:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574856.389534:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574856.389538:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574856.389541:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488824448:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788574856.389549:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488824960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574856.389665:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574856.389804:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574856/real 1788574856] req@ffff94d4049bc380 x1875450488824448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574856.389827:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc380 x1875450488824448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574856.389839:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574856.389843:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574856.389848:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488824448:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574856.389857:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488824576:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574856.389926:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574856.389954:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574856/real 1788574856] req@ffff94d4049bf480 x1875450488824576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574856.389966:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488824576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574856.389974:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574856.389977:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574856.389981:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488824576:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574856.389986:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488824704:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574856.390013:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574856.390032:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574856/real 1788574856] req@ffff94d4049bea00 x1875450488824704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574856.390046:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488824704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574856.390056:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574856.390060:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574856.390063:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488824704:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574856.390070:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488824832:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574856.390093:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574856.390106:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574856/real 1788574856] req@ffff94d4049bce00 x1875450488824832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574856.390119:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488824832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574856.390128:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574856.390132:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574856.390135:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488824832:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574856.390163:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488824960:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574856.390186:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574856.390268:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574856/real 1788574856] req@ffff94d4049bd180 x1875450488824960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574856.390291:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488824960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574856.390299:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574856.390302:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574856.390304:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488824960:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574856.901309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574857.349387:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574857.925280:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574858.373398:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574858.949301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574859.397384:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574859.973308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574860.421395:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574860.997307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574861.445360:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574861.509255:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574861.509264:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574861.509268:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574861.509277:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6009 00000100:00080000:1.0:1788574861.509286:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574861.509292:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574861.509325:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488825088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574861.509349:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574861.509352:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574861.509354:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574861.509356:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6009 00000100:00080000:1.0:1788574861.509359:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574861.509361:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574861.509368:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574861.509377:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488825216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574861.509384:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574861.509386:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574861.509388:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574861.509390:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6009 00000100:00080000:1.0:1788574861.509392:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574861.509398:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574861.509401:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488825088:192.168.202.125@tcp:250:kworker.0 00000008:00100000:1.0:1788574861.509402:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574861.509411:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488825344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574861.509417:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574861.509420:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574861.509421:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574861.509423:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6009 00000100:00080000:1.0:1788574861.509425:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574861.509428:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574861.509432:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574861.509440:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488825472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574861.509447:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574861.509449:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574861.509450:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574861.509452:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6009 00000100:00080000:1.0:1788574861.509455:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574861.509457:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574861.509460:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574861.509466:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574861.509468:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488825600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574861.509501:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574861/real 1788574861] req@ffff94d4049bfb80 x1875450488825088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574861.509517:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488825088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574861.509525:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574861.509529:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574861.509532:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488825088:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574861.509537:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488825216:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574861.509554:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574861.509562:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574861/real 1788574861] req@ffff94d4049bc000 x1875450488825216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574861.509570:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488825216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574861.509576:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574861.509578:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574861.509580:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488825216:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574861.509599:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488825344:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574861.509610:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574861.509616:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574861/real 1788574861] req@ffff94d4049bc700 x1875450488825344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574861.509623:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488825344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574861.509629:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574861.509631:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574861.509633:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488825344:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574861.509636:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488825472:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574861.509647:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574861.509652:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574861/real 1788574861] req@ffff94d4049bed80 x1875450488825472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574861.509664:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488825472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574861.509671:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574861.509674:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574861.509678:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488825472:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574861.509684:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488825600:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574861.509702:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574861.509713:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574861/real 1788574861] req@ffff94d4049bf800 x1875450488825600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574861.509726:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488825600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574861.509731:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574861.509734:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574861.509736:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488825600:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574862.021301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574862.469332:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574863.045365:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574863.493356:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574864.069302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574864.517391:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574865.093329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574865.541429:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574866.117328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574866.565360:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574866.629530:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574866.629557:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574866.629569:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574866.629602:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6014 00000100:00080000:0.0:1788574866.629638:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574866.629659:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574866.629787:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ed80 x1875450488825728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574866.629844:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574866.629851:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574866.629853:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574866.629858:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6014 00000100:00080000:0.0:1788574866.629866:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574866.629871:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574866.629895:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574866.629910:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488825856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574866.629921:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574866.629925:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574866.629927:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574866.629955:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6014 00000100:00080000:0.0:1788574866.629962:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574866.629969:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574866.629979:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574866.630013:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488825984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574866.630025:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574866.630031:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574866.630034:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574866.630039:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6014 00000100:00080000:0.0:1788574866.630043:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574866.630048:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574866.630057:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574866.630077:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488826112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574866.630101:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574866.630111:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574866.630115:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574866.630116:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488825728:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788574866.630126:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6014 00000100:00080000:0.0:1788574866.630141:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574866.630150:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574866.630160:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574866.630181:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488826240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574866.630419:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574866.630495:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574866/real 1788574866] req@ffff94d408d3ed80 x1875450488825728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574866.630519:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ed80 x1875450488825728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574866.630533:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574866.630538:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574866.630542:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488825728:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574866.630555:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488825856:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574866.630580:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574866.630591:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574866/real 1788574866] req@ffff94d408d3d880 x1875450488825856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574866.630600:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488825856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574866.630607:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574866.630610:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574866.630612:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488825856:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574866.630617:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488825984:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574866.630645:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574866.630657:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574866/real 1788574866] req@ffff94d408d3c380 x1875450488825984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574866.630672:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488825984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574866.630690:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574866.630694:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574866.630697:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488825984:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574866.630729:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488826112:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574866.630749:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574866.630757:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574866/real 1788574866] req@ffff94d408d3dc00 x1875450488826112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574866.630766:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488826112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574866.630772:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574866.630775:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574866.630777:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488826112:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574866.630781:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488826240:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574866.630791:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574866.630797:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574866/real 1788574866] req@ffff94d408d3e300 x1875450488826240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574866.630805:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488826240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574866.630811:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574866.630813:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574866.630815:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488826240:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574867.141322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574867.589405:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574868.165287:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574868.613366:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574869.189298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574869.637423:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574870.213314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574870.661371:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574871.237323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574871.685405:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574871.749336:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574871.749346:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574871.749351:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574871.749360:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6020 00000100:00080000:3.0:1788574871.749373:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574871.749381:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574871.749418:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488826368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574871.749451:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574871.749454:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574871.749455:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574871.749458:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6020 00000100:00080000:3.0:1788574871.749461:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574871.749464:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574871.749472:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574871.749485:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488826496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574871.749503:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574871.749508:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574871.749510:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574871.749514:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6020 00000100:00080000:3.0:1788574871.749519:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574871.749523:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574871.749537:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574871.749548:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488826368:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574871.749561:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488826624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574871.749575:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574871.749580:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574871.749591:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574871.749596:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6020 00000100:00080000:3.0:1788574871.749601:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574871.749607:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574871.749612:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574871.749626:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488826752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574871.749637:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574871.749640:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574871.749642:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574871.749648:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6020 00000100:00080000:3.0:1788574871.749653:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788574871.749657:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788574871.749663:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574871.749669:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574871.749684:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488826880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574871.749719:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574871/real 1788574871] req@ffff94d408c57480 x1875450488826368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574871.749744:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488826368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574871.749757:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574871.749763:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574871.749768:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488826368:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574871.749778:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488826496:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574871.749818:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574871.749835:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574871/real 1788574871] req@ffff94d408c54000 x1875450488826496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574871.749849:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488826496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574871.749859:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574871.749863:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574871.749866:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488826496:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574871.749910:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488826624:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574871.749934:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574871.749961:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574871/real 1788574871] req@ffff94d408c55f80 x1875450488826624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574871.749975:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488826624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574871.749985:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574871.749989:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574871.749993:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488826624:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574871.750000:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488826752:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574871.750023:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574871.750034:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574871/real 1788574871] req@ffff94d408c56300 x1875450488826752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574871.750048:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488826752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574871.750058:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574871.750062:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574871.750065:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488826752:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574871.750073:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488826880:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574871.750093:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574871.750104:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574871/real 1788574871] req@ffff94d408c54700 x1875450488826880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574871.750117:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488826880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574871.750126:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574871.750129:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574871.750133:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488826880:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574872.261377:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574872.709447:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574873.285315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574873.733409:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574874.309333:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574874.757432:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574875.333264:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574875.781357:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574876.357344:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574876.805423:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574876.869328:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574876.869339:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574876.869343:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574876.869354:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6025 00000100:00080000:3.0:1788574876.869366:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574876.869374:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574876.869417:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488827008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574876.869446:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574876.869450:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574876.869451:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574876.869461:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6025 00000100:00080000:3.0:1788574876.869464:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574876.869469:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574876.869481:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574876.869495:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488827136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574876.869502:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488827008:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574876.869505:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574876.869510:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574876.869512:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574876.869516:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6025 00000100:00080000:3.0:1788574876.869518:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574876.869520:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574876.869524:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574876.869532:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488827264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574876.869538:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574876.869540:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574876.869541:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574876.869544:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6025 00000100:00080000:3.0:1788574876.869546:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574876.869549:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574876.869551:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574876.869558:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488827392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574876.869563:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574876.869566:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574876.869567:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574876.869570:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6025 00000100:00080000:3.0:1788574876.869573:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574876.869576:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574876.869578:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574876.869585:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488827520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574876.869590:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574876.869637:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574876/real 1788574876] req@ffff94d408c55180 x1875450488827008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574876.869656:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488827008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574876.869668:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574876.869671:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574876.869675:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488827008:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574876.869717:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488827136:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574876.869741:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574876.869754:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574876/real 1788574876] req@ffff94d408c55880 x1875450488827136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574876.869766:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488827136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574876.869781:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574876.869784:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574876.869788:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488827136:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574876.869794:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488827264:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788574876.869812:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574876.869823:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574876/real 1788574876] req@ffff94d408c54380 x1875450488827264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574876.869835:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488827264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574876.869843:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574876.869846:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574876.869849:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488827264:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574876.869855:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488827392:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574876.869885:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574876.869903:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574876/real 1788574876] req@ffff94d408c57800 x1875450488827392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574876.869915:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488827392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574876.869923:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574876.869926:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574876.869930:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488827392:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574876.869936:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488827520:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788574876.869953:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788574876.869963:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574876/real 1788574876] req@ffff94d408c56a00 x1875450488827520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574876.869973:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488827520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574876.869980:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574876.869983:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788574876.869986:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488827520:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574877.381353:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574877.829432:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574878.405353:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574878.853443:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574879.429342:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574879.877442:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574880.453307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574880.901406:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574881.477269:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574881.925418:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574881.989285:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574881.989293:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574881.989296:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574881.989303:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6030 00000100:00080000:3.0:1788574881.989311:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574881.989318:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574881.989348:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488827648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574881.989365:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574881.989367:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574881.989368:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574881.989370:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6030 00000100:00080000:3.0:1788574881.989372:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574881.989375:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574881.989382:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574881.989389:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488827776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574881.989395:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574881.989398:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574881.989399:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574881.989400:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6030 00000100:00080000:3.0:1788574881.989402:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574881.989404:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574881.989406:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574881.989416:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488827904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574881.989421:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574881.989422:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574881.989423:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574881.989425:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6030 00000100:00080000:3.0:1788574881.989427:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574881.989430:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574881.989434:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574881.989444:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488828032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574881.989450:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488827648:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574881.989455:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574881.989461:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574881.989462:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574881.989465:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6030 00000100:00080000:3.0:1788574881.989467:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574881.989468:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574881.989471:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574881.989483:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488828160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574881.989559:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488827776:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574881.989585:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488827904:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788574881.989643:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488828032:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788574881.989681:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488828160:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574882.501286:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574882.949427:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574883.525328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574883.973412:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574884.549302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574884.997436:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788574885.061649:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:1.0:1788574885.061699:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788574885.061716:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788574885.061766:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788574885.061779:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788574885.061788:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788574885.061803:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:3.0:1788574885.061810:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574881/real 1788574885] req@ffff94d408c56d80 x1875450488828032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574885.061836:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488828032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574885.061862:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574885.061871:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574885.061876:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488828032:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574885.061885:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574881/real 1788574885] req@ffff94d408c54a80 x1875450488828160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574885.061899:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488828160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574885.061915:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574885.061919:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574885.061922:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488828160:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574885.061929:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574881/real 1788574885] req@ffff94d408c57100 x1875450488827648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574885.061942:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488827648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574885.061951:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574885.061954:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574885.061957:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488827648:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574885.061964:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574881/real 1788574885] req@ffff94d408c55c00 x1875450488827776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574885.061976:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488827776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574885.061983:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574885.061987:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574885.061990:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488827776:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574885.061996:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574881/real 1788574885] req@ffff94d408c54e00 x1875450488827904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574885.062009:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488827904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574885.062018:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574885.062021:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574885.062025:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488827904:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574885.573302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574886.021351:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574886.597315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574887.045338:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574887.109363:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574887.109372:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574887.109375:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574887.109384:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6035 00000100:00080000:1.0:1788574887.109393:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574887.109400:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574887.109436:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488828288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574887.109454:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574887.109457:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574887.109458:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574887.109460:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6035 00000100:00080000:1.0:1788574887.109463:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574887.109465:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574887.109473:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574887.109481:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488828416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574887.109490:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574887.109493:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574887.109495:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574887.109498:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488828288:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788574887.109501:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6035 00000100:00080000:1.0:1788574887.109508:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574887.109514:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574887.109519:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574887.109534:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488828544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574887.109544:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574887.109548:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574887.109550:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574887.109553:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6035 00000100:00080000:1.0:1788574887.109558:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574887.109562:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574887.109567:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574887.109601:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574887.109603:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488828672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574887.109616:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574887.109620:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574887.109623:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574887.109627:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6035 00000100:00080000:1.0:1788574887.109633:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574887.109637:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574887.109643:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574887/real 1788574887] req@ffff94d4049be680 x1875450488828288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788574887.109647:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574887.109661:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488828800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574887.109686:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488828288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574887.109699:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574887.109704:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574887.109709:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488828288:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574887.109745:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488828416:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574887.109770:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574887.109785:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574887/real 1788574887] req@ffff94d4049bf100 x1875450488828416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574887.109798:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488828416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574887.109814:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574887.109818:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574887.109822:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488828416:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574887.109829:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488828544:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574887.109849:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574887.109861:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574887/real 1788574887] req@ffff94d4049be300 x1875450488828544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574887.109874:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488828544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574887.109883:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574887.109887:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574887.109890:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488828544:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574887.109897:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488828672:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574887.109919:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574887.109931:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574887/real 1788574887] req@ffff94d4049bd500 x1875450488828672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574887.109943:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488828672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574887.109952:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574887.109956:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574887.109960:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488828672:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574887.109967:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488828800:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574887.109986:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574887.109998:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574887/real 1788574887] req@ffff94d4049bdc00 x1875450488828800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574887.110010:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488828800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574887.110019:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574887.110023:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574887.110026:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488828800:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574887.621294:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574888.069314:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574888.645313:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574889.093372:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574889.669286:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574890.117334:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574890.696732:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574891.141348:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574891.717515:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574892.165375:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574892.229356:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574892.229373:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574892.229380:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574892.229394:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6040 00000100:00080000:0.0:1788574892.229409:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574892.229419:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574892.229469:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488828928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574892.229508:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574892.229513:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574892.229515:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574892.229520:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6040 00000100:00080000:0.0:1788574892.229525:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574892.229545:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574892.229558:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574892.229574:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488828928:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574892.229578:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488829056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574892.229599:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574892.229604:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574892.229607:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574892.229613:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6040 00000100:00080000:0.0:1788574892.229618:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574892.229623:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574892.229630:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574892.229647:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488829184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574892.229657:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788574892.229661:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574892.229667:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574892.229670:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574892.229678:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6040 00000100:00080000:0.0:1788574892.229685:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574892.229691:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574892.229698:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574892/real 1788574892] req@ffff94d408d3c700 x1875450488828928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788574892.229699:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574892.229726:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488829312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574892.229728:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488828928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574892.229790:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574892.229797:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574892.229801:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488828928:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788574892.229804:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574892.229810:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574892.229817:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574892.229826:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6040 00000100:00100000:3.0:1788574892.229827:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488829056:192.168.202.125@tcp:38:kworker.0 00000100:00080000:0.0:1788574892.229840:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574892.229850:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:3.0:1788574892.229859:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:0.0:1788574892.229865:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574892.229873:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574892/real 1788574892] req@ffff94d408d3f480 x1875450488829056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574892.229889:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488829440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574892.229889:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488829056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574892.229947:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574892.229954:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574892.229959:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488829056:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574892.229967:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488829184:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574892.229999:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574892.230012:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574892/real 1788574892] req@ffff94d408d3c000 x1875450488829184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574892.230028:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488829184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574892.230037:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574892.230040:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574892.230043:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488829184:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574892.230050:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488829312:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574892.230082:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574892.230101:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574892/real 1788574892] req@ffff94d408d3d180 x1875450488829312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574892.230110:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488829312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574892.230115:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574892.230117:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574892.230119:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488829312:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574892.230131:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488829440:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574892.230146:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574892.230157:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574892/real 1788574892] req@ffff94d408d3df80 x1875450488829440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574892.230171:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488829440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574892.230179:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574892.230183:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574892.230190:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488829440:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574892.741431:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574893.189416:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574893.765640:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574894.213439:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574894.789322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574895.237386:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574895.813840:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574896.261435:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574896.837268:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574897.285365:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574897.349309:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574897.349317:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574897.349321:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574897.349330:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6045 00000100:00080000:0.0:1788574897.349338:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574897.349345:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574897.349378:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488829568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574897.349402:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574897.349405:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574897.349407:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574897.349415:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6045 00000100:00080000:0.0:1788574897.349420:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574897.349424:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574897.349432:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574897.349442:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488829696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574897.349448:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488829568:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788574897.349466:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574897.349470:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574897.349473:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574897.349478:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6045 00000100:00080000:0.0:1788574897.349483:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574897.349487:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574897.349505:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574897.349518:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488829824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574897.349525:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574897.349530:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574897.349532:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788574897.349532:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788574897.349539:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6045 00000100:00080000:0.0:1788574897.349542:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574897.349546:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574897.349550:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574897.349558:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488829952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574897.349565:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574897.349568:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574897.349570:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574897.349578:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6045 00000100:00100000:3.0:1788574897.349579:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574897/real 1788574897] req@ffff94d408d3ce00 x1875450488829568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574897.349584:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574897.349592:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574897.349600:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574897.349608:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488829568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574897.349615:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488830080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574897.349619:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574897.349628:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574897.349635:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488829568:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574897.349657:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488829696:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574897.349676:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574897.349686:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574897/real 1788574897] req@ffff94d408d3d500 x1875450488829696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574897.349694:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488829696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574897.349700:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574897.349702:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574897.349705:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488829696:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574897.349708:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488829824:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574897.349718:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574897.349724:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574897/real 1788574897] req@ffff94d408d3ca80 x1875450488829824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574897.349736:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488829824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574897.349742:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574897.349744:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574897.349746:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488829824:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574897.349749:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488829952:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574897.349759:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574897.349765:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574897/real 1788574897] req@ffff94d408d3ea00 x1875450488829952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574897.349773:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488829952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574897.349780:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574897.349784:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574897.349787:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488829952:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574897.349793:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488830080:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574897.349811:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574897.349821:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574897/real 1788574897] req@ffff94d408d3e680 x1875450488830080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574897.349832:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488830080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574897.349846:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574897.349849:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574897.349852:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488830080:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574897.861387:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574898.309367:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574898.885306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574899.333379:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574899.909332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574900.357450:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574900.933361:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574901.381433:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574901.957321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574902.405438:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574902.469368:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574902.469380:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574902.469385:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574902.469396:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6050 00000100:00080000:0.0:1788574902.469409:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574902.469417:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574902.469482:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f800 x1875450488830208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574902.469523:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574902.469527:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574902.469529:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574902.469533:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6050 00000100:00080000:0.0:1788574902.469536:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574902.469540:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574902.469550:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574902.469563:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f100 x1875450488830336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574902.469572:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574902.469577:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574902.469579:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574902.469585:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488830208:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788574902.469586:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6050 00000100:00080000:0.0:1788574902.469591:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574902.469597:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574902.469602:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574902.469619:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488830464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574902.469630:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574902.469634:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574902.469636:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574902.469641:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6050 00000100:00080000:0.0:1788574902.469646:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574902.469651:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574902.469657:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574902.469669:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488830592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574902.469678:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574902.469681:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574902.469683:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788574902.469688:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788574902.469689:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6050 00000100:00080000:0.0:1788574902.469695:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574902.469698:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574902.469703:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574902.469713:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488830720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574902.469738:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574902/real 1788574902] req@ffff94d408d3f800 x1875450488830208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574902.469760:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f800 x1875450488830208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574902.469770:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574902.469775:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574902.469779:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488830208:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574902.469803:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488830336:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574902.469821:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574902.469830:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574902/real 1788574902] req@ffff94d408d3f100 x1875450488830336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574902.469844:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f100 x1875450488830336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574902.469850:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574902.469852:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574902.469854:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488830336:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574902.469857:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488830464:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574902.469869:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574902.469875:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574902/real 1788574902] req@ffff94d408d3e680 x1875450488830464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574902.469883:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488830464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574902.469888:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574902.469890:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574902.469892:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488830464:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574902.469895:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488830592:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574902.469907:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574902.469912:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574902/real 1788574902] req@ffff94d408d3ea00 x1875450488830592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574902.469920:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488830592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574902.469929:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574902.469931:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574902.469933:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488830592:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574902.469937:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488830720:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574902.469946:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574902.469951:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574902/real 1788574902] req@ffff94d408d3ca80 x1875450488830720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574902.469958:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488830720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574902.469963:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574902.469965:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574902.469967:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488830720:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574902.981322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574903.429397:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574904.005312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574904.453419:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574905.029339:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574905.477348:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574906.053357:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574906.501467:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574907.077298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574907.525344:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574907.589290:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574907.589304:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574907.589311:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574907.589325:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6055 00000100:00080000:0.0:1788574907.589342:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574907.589352:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574907.589406:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488830848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574907.589453:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574907.589457:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574907.589459:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574907.589464:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6055 00000100:00080000:0.0:1788574907.589469:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574907.589475:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574907.589488:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574907.589492:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488830848:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574907.589506:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488830976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574907.589519:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574907.589523:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574907.589525:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574907.589530:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6055 00000100:00080000:0.0:1788574907.589535:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574907.589540:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574907.589545:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574907.589558:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488831104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574907.589565:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574907.589569:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1788574907.589570:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788574907.589572:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574907.589577:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6055 00000100:00080000:0.0:1788574907.589580:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574907.589585:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574907.589589:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574907.589599:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488831232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574907.589605:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574907.589608:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574907.589610:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574907.589614:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574907/real 1788574907] req@ffff94d408d3d500 x1875450488830848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574907.589615:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6055 00000100:00080000:0.0:1788574907.589620:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574907.589626:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574907.589629:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574907.589636:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488831360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574907.589670:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488830848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574907.589682:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574907.589686:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574907.589690:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488830848:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574907.589712:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488830976:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574907.589729:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574907.589738:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574907/real 1788574907] req@ffff94d408d3ce00 x1875450488830976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574907.589747:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488830976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574907.589753:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574907.589755:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574907.589757:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488830976:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574907.589761:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488831104:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574907.589772:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574907.589778:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574907/real 1788574907] req@ffff94d408d3df80 x1875450488831104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574907.589786:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488831104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574907.589796:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574907.589798:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574907.589800:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488831104:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574907.589804:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488831232:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574907.589814:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574907.589820:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574907/real 1788574907] req@ffff94d408d3d180 x1875450488831232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574907.589827:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488831232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574907.589833:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574907.589835:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574907.589837:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488831232:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574907.589840:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488831360:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574907.589850:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574907.589855:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574907/real 1788574907] req@ffff94d408d3c000 x1875450488831360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574907.589863:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488831360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574907.589868:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574907.589870:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574907.589872:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488831360:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574908.101329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574908.549446:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574909.125307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574909.573335:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574910.149277:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574910.597420:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574911.173312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574911.621427:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574912.197297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574912.645429:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574912.709365:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574912.709401:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574912.709406:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574912.709418:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6061 00000100:00080000:0.0:1788574912.709434:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574912.709444:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574912.709497:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488831488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574912.709528:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574912.709531:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574912.709533:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574912.709536:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6061 00000100:00080000:0.0:1788574912.709539:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574912.709542:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574912.709551:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574912.709560:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488831616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574912.709571:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574912.709575:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574912.709577:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574912.709581:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6061 00000100:00080000:0.0:1788574912.709585:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574912.709592:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574912.709593:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488831488:192.168.202.125@tcp:250:kworker.0 00000008:00100000:0.0:1788574912.709599:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574912.709614:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488831744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574912.709634:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574912.709639:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574912.709641:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574912.709644:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6061 00000100:00080000:0.0:1788574912.709647:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574912.709651:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574912.709657:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574912.709666:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488831872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574912.709673:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574912.709676:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574912.709677:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574912.709679:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6061 00000100:00080000:0.0:1788574912.709682:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574912.709684:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574912.709687:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574912.709694:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488832000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574912.709703:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574912.709755:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574912/real 1788574912] req@ffff94d408d3f480 x1875450488831488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574912.709778:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488831488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574912.709789:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574912.709793:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574912.709798:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488831488:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574912.709806:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488831616:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574912.709831:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574912.709846:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574912/real 1788574912] req@ffff94d408d3c700 x1875450488831616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574912.709857:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488831616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574912.709866:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574912.709880:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574912.709884:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488831616:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574912.709916:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488831744:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574912.709935:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574912.709943:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574912/real 1788574912] req@ffff94d408d3e300 x1875450488831744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574912.709952:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488831744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574912.709959:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574912.709961:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574912.709963:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488831744:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574912.709968:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488831872:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574912.709981:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574912.709999:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574912/real 1788574912] req@ffff94d408d3dc00 x1875450488831872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574912.710013:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488831872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574912.710022:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574912.710025:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574912.710028:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488831872:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574912.710035:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488832000:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574912.710058:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574912.710070:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574912/real 1788574912] req@ffff94d408d3c380 x1875450488832000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574912.710084:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488832000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574912.710093:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574912.710096:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574912.710100:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488832000:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574913.221360:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574913.669424:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574914.245274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574914.693400:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574915.269397:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574915.717391:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574916.293306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574916.741426:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574917.317323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574917.765393:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574917.829297:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574917.829306:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574917.829310:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574917.829319:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6066 00000100:00080000:0.0:1788574917.829328:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574917.829361:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574917.829395:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488832128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574917.829422:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574917.829425:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574917.829426:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574917.829429:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6066 00000100:00080000:0.0:1788574917.829432:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574917.829435:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574917.829441:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574917.829450:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ed80 x1875450488832256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574917.829457:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574917.829460:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574917.829461:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574917.829464:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6066 00000100:00080000:0.0:1788574917.829466:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574917.829468:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574917.829471:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574917.829478:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488832128:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574917.829480:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488832384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574917.829497:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574917.829500:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574917.829501:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574917.829503:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6066 00000100:00080000:0.0:1788574917.829505:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574917.829507:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574917.829511:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574917.829519:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488832512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574917.829525:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574917.829527:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574917.829528:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574917.829530:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6066 00000100:00080000:0.0:1788574917.829532:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574917.829535:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574917.829538:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574917.829544:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488832640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574917.829549:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574917.829587:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574917/real 1788574917] req@ffff94d408d3d880 x1875450488832128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574917.829604:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488832128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574917.829611:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574917.829614:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574917.829617:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488832128:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574917.829623:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488832256:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574917.829638:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574917.829646:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574917/real 1788574917] req@ffff94d408d3ed80 x1875450488832256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574917.829658:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ed80 x1875450488832256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574917.829666:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574917.829670:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574917.829673:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488832256:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574917.829698:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488832384:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574917.829717:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574917.829725:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574917/real 1788574917] req@ffff94d408d3c380 x1875450488832384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574917.829734:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488832384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574917.829739:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574917.829747:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574917.829749:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488832384:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574917.829754:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488832512:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574917.829766:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574917.829772:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574917/real 1788574917] req@ffff94d408d3dc00 x1875450488832512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574917.829780:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488832512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574917.829786:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574917.829788:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574917.829790:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488832512:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574917.829793:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488832640:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574917.829802:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574917.829807:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574917/real 1788574917] req@ffff94d408d3e300 x1875450488832640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574917.829815:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488832640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574917.829819:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574917.829821:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574917.829823:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488832640:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574918.341344:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574918.789382:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574919.365320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574919.813377:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574920.389295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574920.837393:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574921.413329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574921.861372:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574922.437353:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574922.885377:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574922.949321:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574922.949331:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574922.949334:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574922.949342:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6071 00000100:00080000:1.0:1788574922.949351:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574922.949357:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574922.949388:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488832768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574922.949414:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574922.949418:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574922.949420:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574922.949425:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6071 00000100:00080000:1.0:1788574922.949428:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574922.949434:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574922.949444:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574922.949458:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd880 x1875450488832896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574922.949459:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488832768:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788574922.949477:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574922.949481:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574922.949482:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574922.949485:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6071 00000100:00080000:1.0:1788574922.949487:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574922.949490:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574922.949493:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574922.949503:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488833024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574922.949510:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574922.949512:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574922.949514:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574922.949516:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6071 00000100:00080000:1.0:1788574922.949518:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574922.949520:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574922.949523:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574922.949530:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488833152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574922.949538:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788574922.949542:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574922.949545:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574922.949546:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574922.949550:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6071 00000100:00080000:1.0:1788574922.949553:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574922.949555:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574922.949558:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574922.949565:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488833280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574922.949582:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574922/real 1788574922] req@ffff94d4049bca80 x1875450488832768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574922.949603:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488832768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574922.949615:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574922.949620:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574922.949624:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488832768:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574922.949646:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488832896:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574922.949669:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574922.949682:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574922/real 1788574922] req@ffff94d4049bd880 x1875450488832896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574922.949696:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd880 x1875450488832896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574922.949705:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574922.949709:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574922.949713:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488832896:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574922.949719:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488833024:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574922.949739:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574922.949750:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574922/real 1788574922] req@ffff94d4049bdc00 x1875450488833024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574922.949770:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488833024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574922.949780:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574922.949783:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574922.949787:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488833024:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574922.949794:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488833152:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574922.949812:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574922.949823:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574922/real 1788574922] req@ffff94d4049bd500 x1875450488833152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574922.949836:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488833152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574922.949844:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574922.949848:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574922.949851:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488833152:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574922.949858:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488833280:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574922.949889:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574922.949901:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574922/real 1788574922] req@ffff94d4049be300 x1875450488833280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574922.949914:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488833280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574922.949931:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574922.949934:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574922.949938:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488833280:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574923.461317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574923.909391:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574924.485278:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574924.933338:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574925.509315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574925.957376:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574926.533337:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574926.981391:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574927.557332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574928.005392:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574928.069365:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574928.069373:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574928.069377:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574928.069386:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6076 00000100:00080000:1.0:1788574928.069395:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574928.069402:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574928.069434:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488833408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574928.069456:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574928.069459:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574928.069461:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574928.069464:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6076 00000100:00080000:1.0:1788574928.069467:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574928.069471:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574928.069478:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574928.069489:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488833536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574928.069498:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574928.069502:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788574928.069505:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488833408:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788574928.069505:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574928.069511:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6076 00000100:00080000:1.0:1788574928.069517:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574928.069521:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574928.069524:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574928.069533:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488833664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574928.069541:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574928.069543:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574928.069545:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574928.069548:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6076 00000100:00080000:1.0:1788574928.069551:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574928.069553:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574928.069557:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574928.069563:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488833792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574928.069569:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574928.069571:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574928.069572:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574928.069574:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6076 00000100:00080000:1.0:1788574928.069577:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574928.069581:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:0.0:1788574928.069584:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:1.0:1788574928.069586:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574928.069597:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488833920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574928.069626:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574928/real 1788574928] req@ffff94d4049bf100 x1875450488833408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574928.069654:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488833408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574928.069665:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574928.069669:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574928.069674:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488833408:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574928.069699:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488833536:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574928.069725:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574928.069740:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574928/real 1788574928] req@ffff94d4049be680 x1875450488833536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574928.069754:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488833536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574928.069762:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574928.069766:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574928.069769:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488833536:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574928.069776:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488833664:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574928.069797:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574928.069811:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574928/real 1788574928] req@ffff94d4049bf800 x1875450488833664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574928.069826:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488833664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574928.069841:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574928.069845:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574928.069849:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488833664:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574928.069867:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488833792:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574928.069894:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574928.069910:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574928/real 1788574928] req@ffff94d4049bed80 x1875450488833792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574928.069936:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488833792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574928.069945:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574928.069950:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574928.069954:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488833792:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574928.069961:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488833920:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574928.069983:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574928.069995:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574928/real 1788574928] req@ffff94d4049bc700 x1875450488833920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574928.070007:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488833920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574928.070015:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574928.070019:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574928.070023:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488833920:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574928.581287:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574929.029355:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574929.605320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574930.053377:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574930.629285:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574931.077420:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574931.653287:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574932.101359:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574932.677314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574933.125380:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574933.189359:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574933.189370:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574933.189375:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574933.189384:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6081 00000100:00080000:0.0:1788574933.189395:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574933.189403:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574933.189452:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488834048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574933.189486:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574933.189490:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574933.189493:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574933.189497:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6081 00000100:00080000:0.0:1788574933.189501:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574933.189506:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574933.189516:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574933.189530:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488834176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574933.189538:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488834048:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788574933.189549:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574933.189554:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574933.189556:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574933.189562:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6081 00000100:00080000:0.0:1788574933.189566:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574933.189571:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574933.189577:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574933.189590:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488834304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574933.189602:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788574933.189607:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788574933.189611:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574933.189615:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574933.189620:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6081 00000100:00080000:0.0:1788574933.189626:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574933.189631:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574933.189637:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574933.189650:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488834432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574933.189660:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574933/real 1788574933] req@ffff94d408d3c700 x1875450488834048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574933.189662:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574933.189671:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574933.189676:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574933.189684:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6081 00000100:00080000:0.0:1788574933.189689:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574933.189695:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488834048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574933.189696:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574933.189705:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788574933.189719:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788574933.189720:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488834560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574933.189729:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574933.189738:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488834048:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574933.189768:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488834176:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574933.189790:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574933.189802:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574933/real 1788574933] req@ffff94d408d3f480 x1875450488834176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574933.189814:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488834176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574933.189823:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574933.189826:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574933.189830:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488834176:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574933.189837:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488834304:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574933.189855:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574933.189866:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574933/real 1788574933] req@ffff94d408d3c000 x1875450488834304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574933.189879:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488834304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574933.189886:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574933.189889:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574933.189892:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488834304:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574933.189907:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488834432:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574933.189924:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574933.189934:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574933/real 1788574933] req@ffff94d408d3d180 x1875450488834432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574933.189947:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488834432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574933.189955:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574933.189959:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574933.189961:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488834432:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574933.189967:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488834560:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574933.189983:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574933.189992:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574933/real 1788574933] req@ffff94d408d3df80 x1875450488834560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574933.190003:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488834560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574933.190011:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574933.190014:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574933.190017:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488834560:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574933.701280:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574934.149365:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574934.725318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574935.173393:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574935.749261:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574936.197355:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574936.773270:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574937.221382:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574937.797348:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574938.245379:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574938.309323:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574938.309335:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574938.309340:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574938.309351:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6086 00000100:00080000:0.0:1788574938.309362:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574938.309369:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574938.309412:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488834688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574938.309443:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574938.309448:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574938.309451:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574938.309456:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6086 00000100:00080000:0.0:1788574938.309461:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574938.309466:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574938.309477:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574938.309488:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488834688:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574938.309497:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488834816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574938.309525:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574938.309528:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574938.309529:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574938.309533:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6086 00000100:00080000:0.0:1788574938.309536:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574938.309538:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574938.309541:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574938.309554:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488834944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574938.309562:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574938.309564:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574938.309565:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574938.309568:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6086 00000100:00080000:0.0:1788574938.309571:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574938.309573:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574938.309576:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574938.309583:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488835072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574938.309588:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574938.309590:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574938.309592:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574938.309595:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6086 00000100:00080000:0.0:1788574938.309597:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574938.309599:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574938.309603:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574938.309613:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488835200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574938.309617:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574938.309689:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574938/real 1788574938] req@ffff94d408d3ce00 x1875450488834688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574938.309717:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488834688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574938.309734:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574938.309739:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574938.309743:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488834688:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574938.309773:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488834816:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574938.309818:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574938.309834:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574938/real 1788574938] req@ffff94d408d3d500 x1875450488834816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574938.309849:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488834816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574938.309857:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574938.309861:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574938.309864:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488834816:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574938.309872:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488834944:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574938.309894:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574938.309906:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574938/real 1788574938] req@ffff94d408d3ca80 x1875450488834944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574938.309926:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488834944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574938.309935:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574938.309938:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574938.309941:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488834944:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574938.309948:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488835072:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574938.309970:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574938.309982:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574938/real 1788574938] req@ffff94d408d3ea00 x1875450488835072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574938.309993:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488835072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574938.309998:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574938.310001:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574938.310003:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488835072:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574938.310008:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488835200:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574938.310023:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574938.310030:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574938/real 1788574938] req@ffff94d408d3e680 x1875450488835200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574938.310038:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488835200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574938.310048:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574938.310050:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574938.310052:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488835200:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574938.821364:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574939.269397:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574939.845301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574940.293353:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574940.869300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574941.317362:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574941.893325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574942.341403:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574942.917331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574943.365370:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574943.429323:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574943.429332:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574943.429336:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574943.429345:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6091 00000100:00080000:0.0:1788574943.429356:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574943.429363:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574943.429397:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f100 x1875450488835328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574943.429426:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574943.429430:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574943.429432:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574943.429435:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6091 00000100:00080000:0.0:1788574943.429438:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574943.429442:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574943.429450:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574943.429463:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f800 x1875450488835456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574943.429473:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574943.429475:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488835328:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788574943.429477:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574943.429482:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574943.429487:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6091 00000100:00080000:0.0:1788574943.429490:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574943.429494:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574943.429500:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574943.429513:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488835584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574943.429522:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574943.429526:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574943.429528:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574943.429531:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6091 00000100:00080000:0.0:1788574943.429536:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788574943.429536:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788574943.429544:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574943.429550:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574943.429562:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488835712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574943.429571:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574943/real 1788574943] req@ffff94d408d3f100 x1875450488835328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574943.429572:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574943.429580:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574943.429583:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574943.429589:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6091 00000100:00080000:0.0:1788574943.429593:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574943.429594:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f100 x1875450488835328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574943.429601:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00080000:3.0:1788574943.429605:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1788574943.429608:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788574943.429610:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574943.429615:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488835328:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574943.429624:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488835840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574943.429644:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488835456:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574943.429662:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574943.429671:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574943/real 1788574943] req@ffff94d408d3f800 x1875450488835456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574943.429683:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f800 x1875450488835456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574943.429688:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574943.429690:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574943.429692:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488835456:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574943.429695:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488835584:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574943.429708:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574943.429713:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574943/real 1788574943] req@ffff94d408d3e680 x1875450488835584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574943.429720:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488835584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574943.429724:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574943.429726:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574943.429727:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488835584:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574943.429730:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488835712:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574943.429739:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574943.429744:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574943/real 1788574943] req@ffff94d408d3ea00 x1875450488835712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574943.429750:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488835712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574943.429758:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574943.429759:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574943.429761:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488835712:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574943.429764:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488835840:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574943.429772:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574943.429776:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574943/real 1788574943] req@ffff94d408d3ca80 x1875450488835840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574943.429782:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488835840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574943.429786:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574943.429787:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574943.429789:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488835840:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574943.941338:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574944.389366:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574944.965307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574945.413355:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574945.989328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574946.437372:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574947.013334:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574947.461350:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574948.037279:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574948.485356:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574948.549347:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574948.549356:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574948.549361:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574948.549393:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6096 00000100:00080000:2.0:1788574948.549402:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574948.549409:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574948.549442:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488835968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574948.549463:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574948.549466:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574948.549467:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574948.549470:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6096 00000100:00080000:2.0:1788574948.549472:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574948.549475:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574948.549481:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574948.549491:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488836096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574948.549498:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574948.549501:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574948.549502:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574948.549506:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6096 00000100:00100000:3.0:1788574948.549508:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488835968:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788574948.549509:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574948.549513:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574948.549520:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574948.549529:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488836224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574948.549535:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574948.549538:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574948.549539:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574948.549542:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6096 00000100:00080000:2.0:1788574948.549545:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574948.549547:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574948.549550:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574948.549557:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488836352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574948.549563:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574948.549566:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574948.549567:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574948.549569:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6096 00000100:00080000:2.0:1788574948.549571:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574948.549574:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574948.549577:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574948.549583:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488836480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574948.549612:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488836096:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574948.549648:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488836224:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574948.549671:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488836352:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574948.549686:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488836480:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574949.061313:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574949.509338:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574950.085273:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574950.533359:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574951.109304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574951.557342:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788574951.621447:0:2392:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:1.0:1788574951.621484:0:2392:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788574951.621511:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788574951.621555:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788574951.621565:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788574951.621575:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788574951.621584:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:3.0:1788574951.621619:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574948/real 1788574951] req@ffff94d408c62680 x1875450488836096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574951.621647:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488836096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574951.621660:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574951.621666:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574951.621671:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488836096:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574951.621680:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574948/real 1788574951] req@ffff94d408c63800 x1875450488836224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574951.621693:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488836224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574951.621701:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574951.621705:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574951.621710:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488836224:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574951.621719:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574948/real 1788574951] req@ffff94d408c60000 x1875450488836352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574951.621739:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488836352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574951.621747:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574951.621751:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574951.621755:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488836352:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574951.621763:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574948/real 1788574951] req@ffff94d408c62d80 x1875450488836480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574951.621775:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488836480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574951.621782:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574951.621785:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574951.621789:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488836480:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574951.621796:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574948/real 1788574951] req@ffff94d408c61180 x1875450488835968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574951.621809:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488835968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574951.621817:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574951.621821:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574951.621825:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488835968:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574952.133329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574952.581364:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574953.157348:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574953.605363:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574953.669338:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574953.669347:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574953.669351:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574953.669359:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6101 00000100:00080000:2.0:1788574953.669369:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574953.669375:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574953.669407:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488836608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574953.669436:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574953.669439:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574953.669441:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574953.669444:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6101 00000100:00080000:2.0:1788574953.669446:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574953.669448:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574953.669455:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574953.669464:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488836736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574953.669471:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574953.669475:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574953.669476:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574953.669480:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6101 00000100:00080000:2.0:1788574953.669484:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574953.669490:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574953.669492:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488836608:192.168.202.125@tcp:250:kworker.0 00000008:00100000:2.0:1788574953.669496:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574953.669513:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488836864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574953.669524:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574953.669527:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574953.669529:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574953.669533:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6101 00000100:00080000:2.0:1788574953.669558:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574953.669563:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:3.0:1788574953.669570:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:2.0:1788574953.669571:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574953.669600:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488836992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574953.669612:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574953.669615:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574953.669617:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574953.669622:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6101 00000100:00080000:2.0:1788574953.669627:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574953.669630:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574953/real 1788574953] req@ffff94d408c62a00 x1875450488836608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574953.669635:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574953.669644:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574953.669661:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488836608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574953.669675:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1788574953.669676:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488837120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574953.669682:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574953.669693:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488836608:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574953.669707:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488836736:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574953.669736:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574953.669750:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574953/real 1788574953] req@ffff94d408c63b80 x1875450488836736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574953.669763:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488836736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574953.669771:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574953.669775:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574953.669778:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488836736:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574953.669804:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488836864:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574953.669827:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574953.669839:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574953/real 1788574953] req@ffff94d408c60a80 x1875450488836864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574953.669852:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488836864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574953.669860:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574953.669863:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574953.669866:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488836864:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574953.669872:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488836992:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574953.669892:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574953.669903:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574953/real 1788574953] req@ffff94d408c63480 x1875450488836992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574953.669914:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488836992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574953.669922:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574953.669925:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574953.669928:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488836992:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574953.669933:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488837120:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574953.669950:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574953.669959:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574953/real 1788574953] req@ffff94d408c61f80 x1875450488837120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574953.669970:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488837120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574953.669978:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574953.669981:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574953.669984:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488837120:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574954.181314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574954.629382:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574955.205300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574955.653328:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574956.229355:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574956.677429:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574957.253431:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574957.701423:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574958.278824:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574958.725452:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574958.789330:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574958.789343:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574958.789348:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574958.789360:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6107 00000100:00080000:0.0:1788574958.789375:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574958.789392:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574958.789441:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488837248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574958.789467:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574958.789470:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574958.789472:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574958.789481:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6107 00000100:00080000:0.0:1788574958.789484:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574958.789488:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574958.789497:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574958.789507:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488837376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574958.789515:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574958.789516:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488837248:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788574958.789519:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574958.789526:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574958.789530:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6107 00000100:00080000:0.0:1788574958.789532:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574958.789535:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574958.789538:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574958.789547:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488837504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574958.789554:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574958.789556:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574958.789557:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574958.789560:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6107 00000100:00080000:0.0:1788574958.789563:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574958.789566:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574958.789569:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574958.789577:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488837632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574958.789582:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574958.789584:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574958.789585:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574958.789588:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6107 00000100:00080000:0.0:1788574958.789591:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574958.789593:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574958.789595:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574958.789602:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574958.789604:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488837760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574958.789655:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574958/real 1788574958] req@ffff94d408d3d500 x1875450488837248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574958.789672:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488837248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574958.789681:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574958.789684:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574958.789687:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488837248:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574958.789709:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488837376:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574958.789724:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574958.789731:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574958/real 1788574958] req@ffff94d408d3ce00 x1875450488837376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574958.789743:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488837376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574958.789748:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574958.789750:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574958.789752:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488837376:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574958.789755:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488837504:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574958.789764:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574958.789768:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574958/real 1788574958] req@ffff94d408d3df80 x1875450488837504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574958.789775:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488837504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574958.789780:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574958.789781:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574958.789783:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488837504:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574958.789786:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488837632:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574958.789795:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574958.789800:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574958/real 1788574958] req@ffff94d408d3d180 x1875450488837632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574958.789807:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488837632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574958.789837:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574958.789840:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574958.789843:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488837632:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574958.789850:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488837760:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574958.789865:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574958.789872:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574958/real 1788574958] req@ffff94d408d3c000 x1875450488837760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574958.789879:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488837760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574958.789884:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574958.789886:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574958.789888:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488837760:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574959.302047:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574959.749449:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574960.325408:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574960.773410:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574961.349701:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574961.797428:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574962.373447:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574962.821419:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574963.397309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574963.845407:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574963.909284:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574963.909295:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574963.909299:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574963.909310:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6112 00000100:00080000:0.0:1788574963.909322:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574963.909329:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574963.909366:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488837888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574963.909386:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574963.909391:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574963.909392:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574963.909396:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6112 00000100:00080000:0.0:1788574963.909398:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574963.909402:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574963.909413:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574963.909431:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488838016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574963.909441:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574963.909445:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574963.909448:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574963.909452:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6112 00000100:00080000:0.0:1788574963.909456:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574963.909459:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574963.909464:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574963.909477:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488838144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574963.909485:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574963.909489:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574963.909491:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574963.909495:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6112 00000100:00080000:0.0:1788574963.909507:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574963.909512:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574963.909519:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574963.909533:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488838272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574963.909543:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574963.909546:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574963.909549:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574963.909553:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6112 00000100:00080000:0.0:1788574963.909557:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574963.909562:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574963.909568:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574963.909580:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488838400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574963.909623:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488837888:192.168.202.125@tcp:250:kworker.0 00000800:00000100:3.0:1788574963.909804:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574963.909845:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574963/real 1788574963] req@ffff94d408d3f480 x1875450488837888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574963.910004:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488837888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574963.910016:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574963.910021:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574963.910284:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488837888:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574963.910292:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488838016:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574963.910318:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574963.910333:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574963/real 1788574963] req@ffff94d408d3c700 x1875450488838016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574963.910346:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488838016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574963.910353:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574963.910357:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574963.910360:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488838016:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574963.910366:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488838144:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574963.910385:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574963.910396:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574963/real 1788574963] req@ffff94d408d3e300 x1875450488838144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574963.910407:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488838144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574963.910421:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574963.910425:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574963.910428:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488838144:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574963.910434:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488838272:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574963.910455:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574963.910465:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574963/real 1788574963] req@ffff94d408d3dc00 x1875450488838272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574963.910476:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488838272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574963.910485:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574963.910488:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574963.910491:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488838272:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574963.910497:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488838400:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574963.910514:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574963.910525:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574963/real 1788574963] req@ffff94d408d3c380 x1875450488838400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574963.910537:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488838400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574963.910544:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574963.910546:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574963.910550:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488838400:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574964.421321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574964.869400:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574965.445275:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574965.893373:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574966.469292:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574966.917404:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574967.493274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574967.941390:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574968.517336:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574968.965380:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574969.029375:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574969.029386:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574969.029390:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574969.029408:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6117 00000100:00080000:0.0:1788574969.029422:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574969.029429:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574969.029479:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ed80 x1875450488838528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574969.029506:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574969.029509:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574969.029511:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574969.029513:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6117 00000100:00080000:0.0:1788574969.029516:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574969.029519:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574969.029528:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574969.029538:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488838656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574969.029545:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574969.029548:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574969.029549:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574969.029551:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6117 00000100:00080000:0.0:1788574969.029553:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574969.029558:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488838528:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788574969.029559:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574969.029562:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574969.029572:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488838784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574969.029578:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574969.029580:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574969.029581:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574969.029583:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6117 00000100:00080000:0.0:1788574969.029586:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574969.029588:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574969.029591:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574969.029597:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488838912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574969.029607:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574969.029609:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574969.029610:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574969.029612:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6117 00000100:00080000:0.0:1788574969.029615:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574969.029617:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574969.029619:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574969.029625:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488839040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574969.029649:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574969.029695:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574969/real 1788574969] req@ffff94d408d3ed80 x1875450488838528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574969.029719:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ed80 x1875450488838528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574969.029727:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574969.029731:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574969.029733:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488838528:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574969.029756:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488838656:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574969.029774:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574969.029782:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574969/real 1788574969] req@ffff94d408d3d880 x1875450488838656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574969.029789:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488838656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574969.029807:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574969.029811:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574969.029814:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488838656:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574969.029821:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488838784:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574969.029839:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574969.029849:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574969/real 1788574969] req@ffff94d408d3c380 x1875450488838784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574969.029857:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488838784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574969.029862:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574969.029864:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574969.029866:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488838784:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574969.029870:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488838912:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574969.029883:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574969.029888:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574969/real 1788574969] req@ffff94d408d3dc00 x1875450488838912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574969.029895:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488838912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574969.029899:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574969.029901:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574969.029902:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488838912:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574969.029905:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488839040:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574969.029915:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574969.029920:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574969/real 1788574969] req@ffff94d408d3e300 x1875450488839040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574969.029927:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488839040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574969.029931:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574969.029933:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574969.029934:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488839040:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574969.541288:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574969.989446:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574970.565276:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574971.013331:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574971.589303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574972.037460:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574972.613322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574973.061477:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574973.637349:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574974.085381:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574974.149329:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574974.149337:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574974.149340:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574974.149349:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6122 00000100:00080000:0.0:1788574974.149359:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574974.149366:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788574974.149397:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488839168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574974.149421:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574974.149424:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574974.149426:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574974.149428:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6122 00000100:00080000:0.0:1788574974.149431:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574974.149433:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574974.149441:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574974.149449:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488839296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574974.149458:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574974.149460:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574974.149462:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574974.149467:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488839168:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788574974.149468:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6122 00000100:00080000:0.0:1788574974.149473:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574974.149490:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574974.149493:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574974.149503:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488839424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574974.149509:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574974.149512:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574974.149513:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574974.149516:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6122 00000100:00080000:0.0:1788574974.149519:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574974.149522:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574974.149525:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574974.149532:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488839552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574974.149538:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574974.149540:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574974.149543:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788574974.149544:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788574974.149548:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6122 00000100:00080000:0.0:1788574974.149552:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574974.149555:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788574974.149559:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574974.149565:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488839680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574974.149589:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574974/real 1788574974] req@ffff94d408d3c700 x1875450488839168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574974.149611:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488839168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574974.149618:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574974.149621:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574974.149624:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488839168:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574974.149648:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488839296:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574974.149684:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574974.149696:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574974/real 1788574974] req@ffff94d408d3f480 x1875450488839296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574974.149706:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488839296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574974.149712:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574974.149714:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574974.149716:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488839296:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574974.149721:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488839424:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574974.149734:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574974.149740:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574974/real 1788574974] req@ffff94d408d3c000 x1875450488839424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574974.149748:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488839424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574974.149753:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574974.149755:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574974.149756:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488839424:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574974.149760:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488839552:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574974.149770:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574974.149776:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574974/real 1788574974] req@ffff94d408d3d180 x1875450488839552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574974.149790:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488839552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574974.149795:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574974.149797:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574974.149799:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488839552:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574974.149802:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488839680:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574974.149812:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574974.149818:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574974/real 1788574974] req@ffff94d408d3df80 x1875450488839680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574974.149825:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488839680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574974.149830:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574974.149832:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574974.149833:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488839680:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574974.661301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574975.109437:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574975.685352:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574976.133461:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574976.709321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574977.157441:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574977.733321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574978.181413:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574978.757300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574979.205387:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574979.269354:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574979.269361:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574979.269365:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574979.269373:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6127 00000100:00080000:1.0:1788574979.269383:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574979.269389:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574979.269431:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488839808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574979.269453:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574979.269456:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574979.269458:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574979.269460:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6127 00000100:00080000:1.0:1788574979.269462:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574979.269465:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574979.269474:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574979.269488:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488839936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574979.269495:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488839808:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788574979.269508:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574979.269513:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574979.269515:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574979.269521:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6127 00000100:00080000:1.0:1788574979.269526:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574979.269531:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574979.269538:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574979.269554:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488840064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574979.269561:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788574979.269573:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574979.269577:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574979.269579:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574979.269585:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6127 00000100:00080000:1.0:1788574979.269590:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574979.269595:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574979/real 1788574979] req@ffff94d4049bc000 x1875450488839808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574979.269596:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574979.269607:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574979.269617:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488839808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574979.269624:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488840192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574979.269629:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574979.269639:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574979.269643:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574979.269645:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574979.269648:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:1.0:1788574979.269652:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6127 00000100:00100000:3.0:1788574979.269656:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488839808:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788574979.269659:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574979.269667:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574979.269672:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574979.269679:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488839936:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788574979.269687:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488840320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574979.269714:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574979.269732:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574979/real 1788574979] req@ffff94d4049bfb80 x1875450488839936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574979.269744:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488839936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574979.269750:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574979.269753:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574979.269755:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488839936:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574979.269759:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488840064:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574979.269772:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574979.269782:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574979/real 1788574979] req@ffff94d4049bd180 x1875450488840064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574979.269801:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488840064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574979.269810:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574979.269814:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574979.269818:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488840064:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574979.269824:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488840192:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574979.269842:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574979.269851:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574979/real 1788574979] req@ffff94d4049bce00 x1875450488840192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574979.269859:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488840192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574979.269864:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574979.269867:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574979.269869:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488840192:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574979.269880:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488840320:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574979.269891:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574979.269897:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574979/real 1788574979] req@ffff94d4049bea00 x1875450488840320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574979.269904:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488840320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574979.269913:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574979.269915:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574979.269917:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488840320:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574979.781308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574980.229403:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574980.805293:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574981.253329:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574981.829270:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574982.277427:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574982.853277:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574983.301355:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574983.877341:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574984.325407:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574984.389337:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574984.389346:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574984.389349:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574984.389357:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6132 00000100:00080000:1.0:1788574984.389367:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574984.389373:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788574984.389410:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488840448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574984.389437:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574984.389440:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574984.389441:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574984.389444:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6132 00000100:00080000:1.0:1788574984.389447:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574984.389450:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574984.389459:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574984.389470:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488840576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574984.389478:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488840448:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788574984.389480:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574984.389486:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574984.389488:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574984.389491:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6132 00000100:00080000:1.0:1788574984.389494:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574984.389496:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574984.389500:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574984.389509:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488840704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574984.389516:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574984.389518:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574984.389519:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574984.389522:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6132 00000100:00080000:1.0:1788574984.389524:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574984.389526:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574984.389530:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574984.389538:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788574984.389539:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488840832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574984.389548:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574984.389551:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574984.389552:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574984.389555:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6132 00000100:00080000:1.0:1788574984.389558:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574984.389560:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788574984.389563:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574984.389571:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488840960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574984.389577:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574984/real 1788574984] req@ffff94d4049bf480 x1875450488840448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574984.389595:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488840448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574984.389603:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574984.389607:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574984.389610:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488840448:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788574984.389629:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488840576:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574984.389644:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574984.389658:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574984/real 1788574984] req@ffff94d4049bc380 x1875450488840576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574984.389671:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc380 x1875450488840576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574984.389677:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574984.389679:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574984.389681:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488840576:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574984.389685:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488840704:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574984.389703:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574984.389710:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574984/real 1788574984] req@ffff94d4049bea00 x1875450488840704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574984.389718:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488840704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574984.389723:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574984.389726:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574984.389728:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488840704:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574984.389731:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488840832:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574984.389742:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574984.389747:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574984/real 1788574984] req@ffff94d4049bce00 x1875450488840832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574984.389755:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488840832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574984.389763:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574984.389766:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574984.389768:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488840832:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574984.389771:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488840960:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574984.389780:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574984.389785:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574984/real 1788574984] req@ffff94d4049bd180 x1875450488840960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574984.389792:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488840960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574984.389797:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574984.389799:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574984.389801:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488840960:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574984.901308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574985.349347:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574985.925328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574986.373367:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574986.949275:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574987.397383:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574987.973335:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574988.421446:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574988.997299:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574989.445440:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574989.509327:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574989.509338:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574989.509343:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574989.509355:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6137 00000100:00080000:2.0:1788574989.509366:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574989.509375:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788574989.509422:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488841088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574989.509456:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574989.509463:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574989.509465:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574989.509469:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6137 00000100:00080000:2.0:1788574989.509473:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574989.509479:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574989.509492:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574989.509503:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488841088:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788574989.509522:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488841216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574989.509539:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574989.509542:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574989.509544:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574989.509548:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6137 00000100:00080000:2.0:1788574989.509568:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574989.509573:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:3.0:1788574989.509579:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:2.0:1788574989.509580:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574989.509598:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60e00 x1875450488841344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574989.509620:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574989.509624:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574989.509627:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574989.509633:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574989/real 1788574989] req@ffff94d408c61500 x1875450488841088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574989.509634:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6137 00000100:00080000:2.0:1788574989.509641:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574989.509649:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788574989.509657:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574989.509676:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63100 x1875450488841472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574989.509685:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488841088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574989.509687:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574989.509695:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574989.509699:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574989.509705:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6137 00000100:00080000:3.0:1788574989.509705:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574989.509711:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574989.509714:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:2.0:1788574989.509719:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574989.509721:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488841088:192.168.202.125@tcp:250:kworker.0 00000008:00100000:2.0:1788574989.509726:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574989.509741:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488841600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574989.509752:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488841216:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574989.509774:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574989.509786:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574989/real 1788574989] req@ffff94d408c61c00 x1875450488841216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574989.509795:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488841216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574989.509810:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574989.509813:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574989.509815:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488841216:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574989.509820:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488841344:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788574989.509833:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574989.509840:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574989/real 1788574989] req@ffff94d408c60e00 x1875450488841344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574989.509848:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60e00 x1875450488841344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574989.509858:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574989.509861:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574989.509863:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488841344:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788574989.509867:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488841472:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574989.509878:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574989.509883:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574989/real 1788574989] req@ffff94d408c63100 x1875450488841472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574989.509896:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63100 x1875450488841472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574989.509904:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574989.509908:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574989.509911:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488841472:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574989.509928:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488841600:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788574989.509950:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574989.509959:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574989/real 1788574989] req@ffff94d408c61880 x1875450488841600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574989.509968:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488841600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574989.509973:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574989.509976:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788574989.509981:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488841600:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574990.021278:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574990.469362:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574991.045358:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574991.493439:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574992.069276:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574992.517397:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574993.093350:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574993.541387:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574994.117371:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574994.565410:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574994.629338:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574994.629350:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574994.629355:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574994.629388:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6142 00000100:00080000:3.0:1788574994.629401:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574994.629409:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574994.629464:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488841728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574994.629495:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574994.629498:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574994.629500:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574994.629502:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6142 00000100:00080000:3.0:1788574994.629506:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574994.629509:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574994.629523:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574994.629534:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488841856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574994.629548:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574994.629552:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574994.629555:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574994.629556:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488841728:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574994.629560:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6142 00000100:00080000:3.0:1788574994.629565:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574994.629570:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574994.629573:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574994.629582:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488841984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574994.629588:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574994.629591:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574994.629593:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574994.629595:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6142 00000100:00080000:3.0:1788574994.629597:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574994.629600:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574994.629603:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574994.629611:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488842112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574994.629618:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574994.629621:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574994.629623:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574994.629625:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6142 00000100:00080000:3.0:1788574994.629628:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574994.629630:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574994.629634:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574994.629640:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488842240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574994.629657:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574994.629713:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574994/real 1788574994] req@ffff94d408c54e00 x1875450488841728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574994.629736:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488841728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574994.629749:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574994.629754:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574994.629759:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488841728:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574994.629788:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488841856:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574994.629816:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574994.629831:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574994/real 1788574994] req@ffff94d408c55c00 x1875450488841856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574994.629845:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488841856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574994.629873:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574994.629877:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574994.629881:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488841856:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574994.629889:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488841984:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574994.629910:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574994.629922:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574994/real 1788574994] req@ffff94d408c57100 x1875450488841984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574994.629935:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488841984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574994.629945:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574994.629948:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574994.629951:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488841984:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574994.629957:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488842112:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574994.629973:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574994.629982:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574994/real 1788574994] req@ffff94d408c54a80 x1875450488842112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574994.629994:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488842112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574994.630002:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574994.630004:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574994.630009:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488842112:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574994.630015:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488842240:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574994.630030:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574994.630038:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574994/real 1788574994] req@ffff94d408c56d80 x1875450488842240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574994.630050:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488842240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574994.630059:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574994.630062:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574994.630065:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488842240:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788574995.141295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574995.589399:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574996.165312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574996.613395:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574997.189301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574997.637376:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574998.213333:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788574998.661413:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574999.237324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788574999.685428:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574999.749272:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574999.749285:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574999.749289:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574999.749298:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6148 00000100:00080000:3.0:1788574999.749307:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574999.749314:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788574999.749349:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488842368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574999.749370:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574999.749373:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574999.749374:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574999.749377:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6148 00000100:00080000:3.0:1788574999.749379:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574999.749382:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574999.749390:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574999.749399:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488842496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574999.749403:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488842368:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788574999.749414:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574999.749417:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574999.749418:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574999.749421:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6148 00000100:00080000:3.0:1788574999.749424:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574999.749426:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574999.749429:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574999.749439:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488842624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574999.749445:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574999.749448:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574999.749449:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574999.749452:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6148 00000100:00080000:3.0:1788574999.749456:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574999.749458:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574999.749461:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788574999.749465:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788574999.749471:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488842752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574999.749478:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574999.749481:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574999.749482:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574999.749485:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6148 00000100:00080000:3.0:1788574999.749488:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574999.749491:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788574999.749494:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574999.749499:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574999/real 1788574999] req@ffff94d408c56680 x1875450488842368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574999.749502:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488842880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574999.749531:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488842368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574999.749540:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574999.749543:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574999.749546:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488842368:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788574999.749565:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488842496:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574999.749584:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574999.749612:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574999/real 1788574999] req@ffff94d408c57b80 x1875450488842496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574999.749622:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488842496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574999.749628:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574999.749631:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574999.749634:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488842496:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574999.749639:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488842624:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788574999.749657:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574999.749664:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574999/real 1788574999] req@ffff94d408c56d80 x1875450488842624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574999.749674:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488842624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574999.749680:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574999.749683:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574999.749686:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488842624:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788574999.749692:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488842752:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574999.749706:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574999.749716:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574999/real 1788574999] req@ffff94d408c54a80 x1875450488842752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574999.749727:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488842752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574999.749734:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574999.749737:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574999.749739:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488842752:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788574999.749744:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488842880:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788574999.749757:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788574999.749763:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574999/real 1788574999] req@ffff94d408c57100 x1875450488842880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574999.749771:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488842880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574999.749776:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574999.749777:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788574999.749779:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488842880:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575000.261270:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575000.709386:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575001.285272:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575001.733426:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575002.309269:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575002.757390:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575003.333252:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575003.781364:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575004.357285:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575004.805387:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575004.869303:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575004.869313:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575004.869318:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575004.869328:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6153 00000100:00080000:3.0:1788575004.869341:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575004.869349:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575004.869393:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488843008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575004.869423:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575004.869433:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575004.869435:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575004.869440:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6153 00000100:00080000:3.0:1788575004.869443:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575004.869449:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575004.869460:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575004.869474:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488843136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575004.869481:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488843008:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575004.869492:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575004.869497:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575004.869498:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575004.869502:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6153 00000100:00080000:3.0:1788575004.869506:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575004.869509:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575004.869514:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575004.869524:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488843264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575004.869533:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575004.869537:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575004.869538:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575004.869542:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6153 00000100:00080000:3.0:1788575004.869546:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575004.869550:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575004.869555:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788575004.869560:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575004.869563:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488843392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575004.869571:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575004.869575:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575004.869577:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575004.869580:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6153 00000100:00080000:3.0:1788575004.869584:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575004.869588:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575004.869592:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575004.869601:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488843520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575004.869615:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575004/real 1788575004] req@ffff94d408c55c00 x1875450488843008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575004.869636:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488843008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575004.869648:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575004.869653:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575004.869657:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488843008:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575004.869682:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488843136:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575004.869702:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575004.869715:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575004/real 1788575004] req@ffff94d408c54e00 x1875450488843136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575004.869729:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488843136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575004.869738:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575004.869742:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575004.869746:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488843136:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575004.869753:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488843264:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575004.869772:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575004.869781:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575004/real 1788575004] req@ffff94d408c56a00 x1875450488843264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575004.869789:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488843264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575004.869795:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575004.869798:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575004.869800:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488843264:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575004.869803:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488843392:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575004.869838:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575004.869846:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575004/real 1788575004] req@ffff94d408c57800 x1875450488843392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575004.869860:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488843392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575004.869868:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575004.869871:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575004.869875:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488843392:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575004.869881:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488843520:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575004.869903:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575004.869915:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575004/real 1788575004] req@ffff94d408c54380 x1875450488843520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575004.869927:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488843520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575004.869936:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575004.869939:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575004.869951:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488843520:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575005.381310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575005.829426:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575006.405325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575006.853390:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575007.429305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575007.877372:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575008.453309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575008.901391:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575009.477288:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575009.925328:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575009.989290:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575009.989297:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575009.989301:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575009.989309:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6158 00000100:00080000:3.0:1788575009.989318:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575009.989324:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575009.989356:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488843648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575009.989382:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575009.989384:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575009.989385:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575009.989388:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6158 00000100:00080000:3.0:1788575009.989391:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575009.989394:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575009.989401:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575009.989411:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488843776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575009.989417:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575009.989420:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575009.989421:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575009.989423:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6158 00000100:00080000:3.0:1788575009.989425:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575009.989427:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575009.989430:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575009.989437:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488843904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575009.989445:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488843648:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575009.989446:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575009.989451:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575009.989454:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575009.989459:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6158 00000100:00080000:3.0:1788575009.989461:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575009.989463:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575009.989466:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575009.989474:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488844032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575009.989490:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575009.989493:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575009.989495:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575009.989500:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6158 00000100:00080000:3.0:1788575009.989504:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575009.989508:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:0.0:1788575009.989513:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:3.0:1788575009.989516:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575009.989532:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488844160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575009.989548:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575009/real 1788575009] req@ffff94d408c55880 x1875450488843648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575009.989565:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488843648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575009.989578:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575009.989581:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575009.989585:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488843648:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575009.989590:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488843776:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575009.989603:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575009.989611:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575009/real 1788575009] req@ffff94d408c55180 x1875450488843776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575009.989618:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488843776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575009.989624:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575009.989626:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575009.989628:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488843776:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575009.989646:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488843904:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575009.989656:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575009.989661:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575009/real 1788575009] req@ffff94d408c54700 x1875450488843904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575009.989669:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488843904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575009.989674:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575009.989676:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575009.989678:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488843904:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575009.989682:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488844032:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575009.989693:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575009.989699:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575009/real 1788575009] req@ffff94d408c56300 x1875450488844032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575009.989706:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488844032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575009.989711:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575009.989713:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575009.989715:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488844032:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575009.989718:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488844160:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575009.989727:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575009.989733:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575009/real 1788575009] req@ffff94d408c55f80 x1875450488844160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575009.989750:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488844160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575009.989755:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575009.989757:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575009.989759:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488844160:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575010.501318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575010.949364:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575011.525300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575011.973329:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575012.549293:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575012.997362:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575013.573291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575014.021367:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575014.597298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575015.045355:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575015.109341:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575015.109354:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575015.109359:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575015.109371:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6163 00000100:00080000:2.0:1788575015.109382:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575015.109391:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575015.109426:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488844288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575015.109445:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575015.109448:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575015.109450:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575015.109452:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6163 00000100:00080000:2.0:1788575015.109455:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575015.109459:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575015.109467:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575015.109476:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63100 x1875450488844416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575015.109481:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488844288:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788575015.109488:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575015.109492:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575015.109493:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575015.109497:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6163 00000100:00080000:2.0:1788575015.109499:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575015.109502:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575015.109505:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575015.109514:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60e00 x1875450488844544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575015.109520:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575015.109523:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575015.109524:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575015.109527:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6163 00000100:00080000:2.0:1788575015.109530:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575015.109533:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575015.109536:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575015.109544:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488844672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575015.109555:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575015.109557:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575015.109559:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575015.109562:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6163 00000100:00080000:2.0:1788575015.109565:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575015.109568:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575015.109571:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575015.109578:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488844416:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575015.109581:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488844800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575015.109618:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488844544:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575015.109634:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488844672:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575015.109651:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488844800:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575015.621290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575016.069352:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575016.645319:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575017.094535:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575017.669327:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575018.117374:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788575018.181528:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:1.0:1788575018.181575:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788575018.181594:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575018.181640:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575018.181651:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575018.181668:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575018.181681:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:0.0:1788575018.181691:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575015/real 1788575018] req@ffff94d408c63100 x1875450488844416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575018.181714:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63100 x1875450488844416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575018.181724:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575018.181737:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575018.181741:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488844416:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575018.181750:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575015/real 1788575018] req@ffff94d408c60e00 x1875450488844544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575018.181766:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60e00 x1875450488844544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575018.181775:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575018.181779:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575018.181782:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488844544:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575018.181789:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575015/real 1788575018] req@ffff94d408c61c00 x1875450488844672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575018.181802:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488844672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575018.181811:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575018.181815:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575018.181825:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488844672:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575018.181834:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575015/real 1788575018] req@ffff94d408c61880 x1875450488844288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575018.181845:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488844288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575018.181851:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575018.181853:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575018.181855:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488844288:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575018.181876:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575015/real 1788575018] req@ffff94d408c61500 x1875450488844800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575018.181884:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488844800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575018.181889:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575018.181891:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575018.181893:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488844800:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575018.693353:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575019.141387:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575019.717329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575020.165340:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575020.229338:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575020.229359:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575020.229363:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575020.229375:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6168 00000100:00080000:3.0:1788575020.229386:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575020.229394:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575020.229433:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488844928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575020.229468:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575020.229472:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575020.229475:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575020.229480:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6168 00000100:00080000:3.0:1788575020.229484:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575020.229489:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575020.229498:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575020.229513:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488845056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575020.229526:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488844928:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575020.229529:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575020.229537:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575020.229539:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575020.229544:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6168 00000100:00080000:3.0:1788575020.229549:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575020.229554:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575020.229560:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575020.229575:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488845184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575020.229586:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575020.229590:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575020.229592:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788575020.229599:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575020.229604:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6168 00000100:00080000:3.0:1788575020.229610:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575020.229615:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575020.229622:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575020.229637:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488845312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575020.229646:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575020/real 1788575020] req@ffff94d408c54000 x1875450488844928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575020.229651:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575020.229661:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575020.229664:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575020.229674:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6168 00000100:00080000:3.0:1788575020.229681:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575020.229687:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575020.229693:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575020.229694:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488844928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575020.229710:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488845440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575020.229711:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575020.229720:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575020.229729:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488844928:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575020.229777:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488845056:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575020.229802:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575020.229815:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575020/real 1788575020] req@ffff94d408c57480 x1875450488845056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575020.229828:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488845056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575020.229845:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575020.229848:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575020.229852:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488845056:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575020.229858:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488845184:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575020.229879:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575020.229890:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575020/real 1788575020] req@ffff94d408c55f80 x1875450488845184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575020.229901:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488845184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575020.229915:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575020.229918:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575020.229921:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488845184:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575020.229926:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488845312:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575020.229946:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575020.229958:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575020/real 1788575020] req@ffff94d408c56300 x1875450488845312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575020.229970:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488845312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575020.229979:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575020.229982:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575020.229986:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488845312:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575020.229993:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488845440:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575020.230014:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575020.230026:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575020/real 1788575020] req@ffff94d408c54700 x1875450488845440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575020.230038:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488845440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575020.230046:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575020.230050:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575020.230054:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488845440:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575020.741343:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575021.189430:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575021.765320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575022.213378:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575022.789386:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575023.237378:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575023.813620:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575024.261394:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575024.837315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575025.285397:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575025.349327:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575025.349359:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575025.349362:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575025.349370:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6173 00000100:00080000:2.0:1788575025.349380:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575025.349387:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575025.349419:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488845568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575025.349441:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575025.349444:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575025.349445:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575025.349448:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6173 00000100:00080000:2.0:1788575025.349450:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575025.349452:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575025.349460:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575025.349470:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488845696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575025.349478:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575025.349482:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488845568:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788575025.349482:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575025.349486:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575025.349494:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6173 00000100:00080000:2.0:1788575025.349496:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575025.349498:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575025.349501:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575025.349511:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488845824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575025.349518:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575025.349521:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575025.349523:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575025.349528:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6173 00000100:00080000:2.0:1788575025.349532:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575025.349537:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575025.349543:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788575025.349555:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575025.349558:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488845952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575025.349570:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575025.349574:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575025.349577:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575025.349582:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6173 00000100:00080000:2.0:1788575025.349586:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788575025.349590:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575025/real 1788575025] req@ffff94d408c61f80 x1875450488845568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575025.349594:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575025.349602:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575025.349613:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488845568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575025.349614:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488846080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575025.349639:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575025.349645:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575025.349649:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488845568:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575025.349669:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488845696:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575025.349685:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575025.349693:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575025/real 1788575025] req@ffff94d408c63480 x1875450488845696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575025.349701:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488845696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575025.349706:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575025.349708:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575025.349711:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488845696:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575025.349714:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488845824:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575025.349724:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575025.349730:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575025/real 1788575025] req@ffff94d408c60a80 x1875450488845824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575025.349737:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488845824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575025.349742:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575025.349744:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575025.349746:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488845824:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575025.349749:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488845952:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575025.349765:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575025.349771:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575025/real 1788575025] req@ffff94d408c63b80 x1875450488845952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575025.349778:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488845952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575025.349783:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575025.349785:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575025.349787:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488845952:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575025.349791:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488846080:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575025.349800:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575025.349805:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575025/real 1788575025] req@ffff94d408c62a00 x1875450488846080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575025.349812:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488846080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575025.349817:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575025.349819:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575025.349821:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488846080:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575025.861597:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575026.309324:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575026.885353:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575027.333362:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575027.909345:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575028.357460:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575028.933409:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575029.381344:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575029.957314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575030.405370:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575030.469331:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575030.469339:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575030.469344:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575030.469354:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6178 00000100:00080000:3.0:1788575030.469365:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575030.469373:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575030.469422:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488846208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575030.469449:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575030.469465:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575030.469466:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575030.469470:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6178 00000100:00080000:3.0:1788575030.469474:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575030.469478:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575030.469487:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575030.469500:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488846208:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575030.469501:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488846336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575030.469516:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575030.469519:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575030.469520:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575030.469523:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6178 00000100:00080000:3.0:1788575030.469526:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575030.469530:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575030.469535:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575030.469549:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488846464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575030.469561:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575030.469565:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575030.469567:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788575030.469573:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575030.469574:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6178 00000100:00080000:3.0:1788575030.469590:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575030.469599:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575030.469606:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575030.469609:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575030/real 1788575030] req@ffff94d408c55180 x1875450488846208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575030.469625:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488846592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575030.469627:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488846208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575030.469658:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575030.469661:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575030.469662:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575030.469669:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488846208:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575030.469670:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575030.469675:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575030.469681:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6178 00000100:00080000:3.0:1788575030.469685:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575030.469690:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575030.469692:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488846336:192.168.202.125@tcp:38:kworker.0 00000008:00100000:3.0:1788575030.469701:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788575030.469718:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575030.469719:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488846720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575030.469730:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575030/real 1788575030] req@ffff94d408c55880 x1875450488846336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575030.469741:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488846336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575030.469748:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575030.469751:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575030.469754:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488846336:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575030.469758:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488846464:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575030.469769:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575030.469775:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575030/real 1788575030] req@ffff94d408c54380 x1875450488846464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575030.469783:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488846464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575030.469789:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575030.469790:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575030.469792:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488846464:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575030.469796:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488846592:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575030.469806:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575030.469811:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575030/real 1788575030] req@ffff94d408c57800 x1875450488846592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575030.469819:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488846592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575030.469824:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575030.469826:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575030.469827:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488846592:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575030.469837:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488846720:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575030.469847:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575030.469853:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575030/real 1788575030] req@ffff94d408c56a00 x1875450488846720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575030.469861:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488846720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575030.469866:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575030.469867:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575030.469869:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488846720:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575030.981304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575031.429376:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575032.005332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575032.453441:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575033.029276:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575033.477380:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575034.053300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575034.501380:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575035.077276:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575035.525332:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575035.589306:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575035.589317:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575035.589322:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575035.589334:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6183 00000100:00080000:3.0:1788575035.589343:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575035.589350:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575035.589380:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488846848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575035.589409:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575035.589412:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575035.589414:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575035.589417:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6183 00000100:00080000:3.0:1788575035.589420:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575035.589423:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575035.589431:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575035.589442:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488846976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575035.589452:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575035.589458:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488846848:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575035.589458:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575035.589462:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575035.589468:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6183 00000100:00080000:3.0:1788575035.589471:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575035.589474:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575035.589479:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575035.589489:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488847104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575035.589496:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575035.589499:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575035.589500:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575035.589503:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6183 00000100:00080000:3.0:1788575035.589507:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575035.589509:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575035.589513:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788575035.589520:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575035.589523:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488847232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575035.589532:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575035.589534:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575035.589536:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575035.589539:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6183 00000100:00080000:3.0:1788575035.589543:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575035.589546:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575035.589567:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575035/real 1788575035] req@ffff94d408c54e00 x1875450488846848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575035.589600:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488846848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788575035.589615:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788575035.589620:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575035.589624:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575035.589631:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488846848:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575035.589633:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488847360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575035.589656:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488846976:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575035.589677:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575035.589687:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575035/real 1788575035] req@ffff94d408c55c00 x1875450488846976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575035.589698:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488846976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575035.589704:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575035.589706:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575035.589708:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488846976:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575035.589712:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488847104:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575035.589722:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575035.589728:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575035/real 1788575035] req@ffff94d408c57100 x1875450488847104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575035.589736:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488847104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575035.589742:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575035.589744:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575035.589746:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488847104:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575035.589750:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488847232:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575035.589761:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575035.589768:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575035/real 1788575035] req@ffff94d408c54a80 x1875450488847232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575035.589775:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488847232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575035.589781:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575035.589782:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575035.589784:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488847232:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575035.589788:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488847360:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575035.589799:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575035.589805:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575035/real 1788575035] req@ffff94d408c56d80 x1875450488847360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575035.589813:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488847360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575035.589818:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575035.589821:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575035.589823:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488847360:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575036.101312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575036.549388:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575037.125267:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575037.573359:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575038.149298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575038.597349:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575039.173299:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575039.621322:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575040.197318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575040.645345:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575040.709281:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575040.709288:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575040.709292:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575040.709300:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6189 00000100:00080000:0.0:1788575040.709308:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575040.709313:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575040.709342:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488847488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575040.709360:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575040.709363:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575040.709364:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575040.709366:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6189 00000100:00080000:0.0:1788575040.709368:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575040.709370:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575040.709376:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575040.709384:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488847616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575040.709395:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575040.709398:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575040.709400:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575040.709401:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488847488:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788575040.709405:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6189 00000100:00080000:0.0:1788575040.709410:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575040.709412:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575040.709415:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575040.709427:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488847744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575040.709435:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575040.709438:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575040.709440:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575040.709443:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6189 00000100:00080000:0.0:1788575040.709448:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575040.709451:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:1.0:1788575040.709455:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:0.0:1788575040.709458:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575040.709471:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488847872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575040.709478:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575040.709480:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575040.709481:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575040.709484:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6189 00000100:00100000:1.0:1788575040.709488:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575040/real 1788575040] req@ffff94d408d3ce00 x1875450488847488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575040.709489:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575040.709496:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575040.709503:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575040.709509:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488847488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575040.709512:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488848000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575040.709528:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575040.709532:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575040.709535:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488847488:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575040.709552:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488847616:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575040.709565:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575040.709572:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575040/real 1788575040] req@ffff94d408d3d500 x1875450488847616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575040.709579:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488847616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575040.709584:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575040.709585:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575040.709587:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488847616:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575040.709590:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488847744:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575040.709599:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575040.709604:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575040/real 1788575040] req@ffff94d408d3ca80 x1875450488847744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575040.709610:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488847744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575040.709615:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575040.709617:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575040.709618:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488847744:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575040.709621:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488847872:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575040.709629:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575040.709634:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575040/real 1788575040] req@ffff94d408d3ea00 x1875450488847872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575040.709640:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488847872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575040.709644:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575040.709646:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575040.709648:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488847872:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575040.709650:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488848000:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575040.709658:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575040.709663:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575040/real 1788575040] req@ffff94d408d3e680 x1875450488848000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575040.709668:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488848000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575040.709676:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575040.709678:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575040.709680:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488848000:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575041.221286:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575041.669335:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575042.245296:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575042.693327:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575043.269310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575043.717331:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575044.293330:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575044.741367:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575045.317274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575045.765328:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575045.829306:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575045.829313:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575045.829317:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575045.829325:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6194 00000100:00080000:3.0:1788575045.829334:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575045.829339:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575045.829367:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488848128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575045.829385:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575045.829388:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575045.829389:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575045.829392:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6194 00000100:00080000:3.0:1788575045.829393:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575045.829396:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575045.829402:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575045.829410:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488848256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575045.829417:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575045.829419:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575045.829421:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575045.829424:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488848128:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575045.829426:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6194 00000100:00080000:3.0:1788575045.829431:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575045.829435:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575045.829438:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575045.829446:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488848384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575045.829451:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575045.829453:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575045.829454:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575045.829456:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6194 00000100:00080000:3.0:1788575045.829458:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575045.829460:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575045.829462:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575045.829468:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488848512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575045.829473:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575045.829475:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575045.829476:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575045.829478:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6194 00000100:00080000:3.0:1788575045.829488:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575045.829490:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575045.829493:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788575045.829495:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575045.829502:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488848640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575045.829539:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575045/real 1788575045] req@ffff94d408c57b80 x1875450488848128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575045.829556:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488848128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575045.829565:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575045.829568:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575045.829571:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488848128:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575045.829592:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488848256:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575045.829614:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575045.829625:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575045/real 1788575045] req@ffff94d408c56680 x1875450488848256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575045.829637:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488848256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575045.829654:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575045.829658:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575045.829661:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488848256:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575045.829666:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488848384:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575045.829685:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575045.829696:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575045/real 1788575045] req@ffff94d408c56d80 x1875450488848384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575045.829707:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488848384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575045.829714:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575045.829717:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575045.829720:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488848384:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575045.829725:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488848512:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575045.829742:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575045.829751:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575045/real 1788575045] req@ffff94d408c54a80 x1875450488848512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575045.829762:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488848512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575045.829769:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575045.829772:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575045.829775:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488848512:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575045.829780:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488848640:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575045.829802:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575045.829826:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575045/real 1788575045] req@ffff94d408c57100 x1875450488848640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575045.829838:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488848640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575045.829845:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575045.829848:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575045.829851:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488848640:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575046.341279:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575046.789362:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575047.365301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575047.813315:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575048.389310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575048.837357:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575049.413312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575049.861378:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575050.437265:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575050.885329:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575050.949343:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575050.949353:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575050.949357:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575050.949366:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6199 00000100:00080000:0.0:1788575050.949376:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575050.949381:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575050.949411:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f800 x1875450488848768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575050.949433:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575050.949436:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575050.949437:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575050.949439:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6199 00000100:00080000:0.0:1788575050.949441:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575050.949443:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575050.949450:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575050.949457:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f100 x1875450488848896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575050.949464:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575050.949468:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575050.949469:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575050.949471:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6199 00000100:00080000:0.0:1788575050.949475:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575050.949476:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488848768:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788575050.949479:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575050.949484:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575050.949492:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488849024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575050.949498:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575050.949500:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575050.949501:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575050.949503:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6199 00000100:00080000:0.0:1788575050.949505:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575050.949507:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575050.949510:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575050.949516:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488849152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575050.949521:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575050.949523:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575050.949525:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575050.949528:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6199 00000100:00080000:0.0:1788575050.949533:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575050.949537:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575050.949542:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788575050.949547:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575050.949555:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488849280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575050.949579:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575050/real 1788575050] req@ffff94d408d3f800 x1875450488848768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575050.949592:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f800 x1875450488848768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575050.949599:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575050.949602:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575050.949605:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488848768:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575050.949609:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488848896:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575050.949624:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575050.949630:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575050/real 1788575050] req@ffff94d408d3f100 x1875450488848896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575050.949637:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f100 x1875450488848896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575050.949652:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575050.949655:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575050.949657:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488848896:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575050.949676:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488849024:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575050.949695:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575050.949706:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575050/real 1788575050] req@ffff94d408d3e680 x1875450488849024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575050.949717:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488849024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575050.949723:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575050.949725:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575050.949727:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488849024:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575050.949730:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488849152:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575050.949743:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575050.949750:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575050/real 1788575050] req@ffff94d408d3ea00 x1875450488849152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575050.949756:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488849152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575050.949760:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575050.949762:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575050.949763:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488849152:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575050.949766:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488849280:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575050.949774:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575050.949779:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575050/real 1788575050] req@ffff94d408d3ca80 x1875450488849280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575050.949785:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488849280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575050.949789:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575050.949791:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575050.949792:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488849280:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575051.461326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575051.909330:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575052.485267:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575052.933319:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575053.509294:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575053.957355:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575054.533256:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575054.981384:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575055.557285:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575056.005392:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575056.069282:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575056.069290:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575056.069293:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575056.069302:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6204 00000100:00080000:0.0:1788575056.069310:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575056.069316:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575056.069347:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488849408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575056.069369:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575056.069372:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575056.069373:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575056.069376:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6204 00000100:00080000:0.0:1788575056.069378:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575056.069380:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575056.069387:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575056.069395:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488849536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575056.069402:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575056.069404:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575056.069406:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575056.069408:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6204 00000100:00080000:0.0:1788575056.069411:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575056.069415:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488849408:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788575056.069418:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575056.069426:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575056.069440:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488849664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575056.069450:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575056.069454:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575056.069456:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575056.069460:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6204 00000100:00080000:0.0:1788575056.069465:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575056.069470:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:1.0:1788575056.069475:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:0.0:1788575056.069477:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575056.069490:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488849792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575056.069497:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575056.069500:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575056.069501:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575056.069504:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6204 00000100:00080000:0.0:1788575056.069508:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575056.069509:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575056/real 1788575056] req@ffff94d408d3d500 x1875450488849408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575056.069515:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575056.069520:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575056.069526:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488849920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575056.069541:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488849408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575056.069550:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575056.069553:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575056.069557:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488849408:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575056.069563:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488849536:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575056.069584:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575056.069592:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575056/real 1788575056] req@ffff94d408d3ce00 x1875450488849536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575056.069602:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488849536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575056.069611:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575056.069614:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575056.069618:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488849536:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575056.069642:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488849664:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575056.069664:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575056.069674:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575056/real 1788575056] req@ffff94d408d3df80 x1875450488849664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575056.069682:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488849664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575056.069699:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575056.069701:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575056.069703:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488849664:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575056.069707:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488849792:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575056.069720:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575056.069726:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575056/real 1788575056] req@ffff94d408d3d180 x1875450488849792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575056.069733:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488849792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575056.069738:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575056.069740:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575056.069742:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488849792:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575056.069746:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488849920:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575056.069755:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575056.069760:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575056/real 1788575056] req@ffff94d408d3c000 x1875450488849920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575056.069767:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488849920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575056.069772:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575056.069774:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575056.069776:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488849920:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575056.581258:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575057.029357:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575057.605305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575058.053416:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575058.629281:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575059.077322:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575059.653308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575060.101352:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575060.677301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575061.125375:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575061.189336:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575061.189344:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575061.189347:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575061.189356:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6209 00000100:00080000:2.0:1788575061.189365:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575061.189371:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575061.189403:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488850048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575061.189428:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575061.189430:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575061.189432:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575061.189435:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6209 00000100:00080000:2.0:1788575061.189437:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575061.189439:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575061.189446:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575061.189455:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488850176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575061.189467:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575061.189469:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575061.189471:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575061.189474:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6209 00000100:00080000:2.0:1788575061.189477:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575061.189479:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575061.189483:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575061.189496:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488850304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575061.189497:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488850048:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788575061.189522:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575061.189524:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575061.189525:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575061.189528:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6209 00000100:00080000:2.0:1788575061.189530:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575061.189533:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575061.189537:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575061.189549:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488850432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575061.189556:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575061.189558:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575061.189559:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575061.189561:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6209 00000100:00080000:2.0:1788575061.189565:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575061.189567:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575061.189571:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575061.189580:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488850560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575061.189610:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575061.189651:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575061/real 1788575061] req@ffff94d408c61180 x1875450488850048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575061.189668:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488850048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575061.189677:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575061.189681:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575061.189684:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488850048:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575061.189690:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488850176:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575061.189706:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575061.189714:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575061/real 1788575061] req@ffff94d408c62d80 x1875450488850176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575061.189723:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488850176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575061.189729:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575061.189731:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575061.189734:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488850176:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575061.189758:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488850304:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575061.189770:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575061.189776:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575061/real 1788575061] req@ffff94d408c60000 x1875450488850304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575061.189785:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488850304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575061.189791:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575061.189793:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575061.189795:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488850304:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575061.189799:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488850432:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575061.189809:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575061.189815:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575061/real 1788575061] req@ffff94d408c63800 x1875450488850432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575061.189822:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488850432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575061.189827:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575061.189830:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575061.189831:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488850432:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575061.189835:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488850560:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575061.189845:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575061.189851:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575061/real 1788575061] req@ffff94d408c62680 x1875450488850560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575061.189859:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488850560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575061.189864:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575061.189866:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575061.189868:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488850560:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575061.701302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575062.149345:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575062.725346:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575063.173382:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575063.749335:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575064.197392:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575064.773321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575065.221356:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575065.797248:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575066.245408:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575066.309345:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575066.309354:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575066.309358:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575066.309369:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6214 00000100:00080000:3.0:1788575066.309380:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575066.309388:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575066.309421:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488850688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575066.309445:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575066.309448:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575066.309450:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575066.309453:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6214 00000100:00080000:3.0:1788575066.309455:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575066.309457:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575066.309465:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575066.309473:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488850816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575066.309480:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575066.309484:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575066.309485:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575066.309487:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6214 00000100:00080000:3.0:1788575066.309489:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575066.309492:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575066.309494:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575066.309504:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488850944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575066.309505:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488850688:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575066.309520:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575066.309523:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575066.309524:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575066.309526:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6214 00000100:00080000:3.0:1788575066.309529:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575066.309532:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575066.309536:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575066.309544:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488851072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575066.309551:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575066.309554:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575066.309555:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575066.309557:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6214 00000100:00080000:3.0:1788575066.309560:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575066.309562:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575066.309565:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575066.309573:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488851200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575066.309580:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575066.309618:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575066/real 1788575066] req@ffff94d408c55c00 x1875450488850688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575066.309635:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488850688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575066.309645:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575066.309649:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575066.309653:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488850688:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575066.309660:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488850816:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575066.309681:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575066.309692:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575066/real 1788575066] req@ffff94d408c54e00 x1875450488850816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575066.309703:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488850816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575066.309711:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575066.309714:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575066.309718:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488850816:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575066.309740:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488850944:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575066.309759:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575066.309772:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575066/real 1788575066] req@ffff94d408c56a00 x1875450488850944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575066.309785:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488850944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575066.309794:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575066.309798:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575066.309801:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488850944:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575066.309808:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488851072:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575066.309827:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575066.309837:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575066/real 1788575066] req@ffff94d408c57800 x1875450488851072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575066.309850:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488851072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575066.309858:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575066.309862:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575066.309865:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488851072:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575066.309871:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488851200:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575066.309891:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575066.309902:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575066/real 1788575066] req@ffff94d408c54380 x1875450488851200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575066.309915:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488851200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575066.309924:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575066.309927:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575066.309930:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488851200:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575066.821282:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575067.269348:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575067.845269:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575068.293439:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575068.869301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575069.317473:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575069.893287:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575070.341346:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575070.917267:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575071.365362:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575071.429302:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575071.429310:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575071.429314:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575071.429322:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6219 00000100:00080000:3.0:1788575071.429331:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575071.429337:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575071.429366:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488851328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575071.429407:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575071.429415:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575071.429417:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575071.429423:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6219 00000100:00080000:3.0:1788575071.429427:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575071.429431:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575071.429441:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575071.429457:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488851456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575071.429465:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488851328:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575071.429471:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575071.429477:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575071.429479:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575071.429484:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6219 00000100:00080000:3.0:1788575071.429489:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575071.429493:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575071.429498:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575071.429512:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488851584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575071.429522:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575071.429525:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575071.429527:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575071.429532:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6219 00000100:00080000:3.0:1788575071.429537:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788575071.429538:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575071.429547:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575071.429554:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575071.429566:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488851712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575071.429575:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575071/real 1788575071] req@ffff94d408c55880 x1875450488851328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575071.429578:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575071.429588:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575071.429593:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575071.429599:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6219 00000100:00100000:0.0:1788575071.429602:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488851328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575071.429612:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575071.429617:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575071.429620:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488851328:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575071.429665:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575071.429673:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575071.429676:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488851456:192.168.202.125@tcp:38:kworker.0 00000008:00100000:3.0:1788575071.429682:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788575071.429702:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575071.429716:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575071/real 1788575071] req@ffff94d408c55180 x1875450488851456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575071.429725:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488851456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575071.429731:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575071.429733:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575071.429735:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488851456:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575071.429739:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488851584:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575071.429753:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575071.429760:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575071/real 1788575071] req@ffff94d408c54700 x1875450488851584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575071.429769:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488851584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575071.429776:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575071.429778:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575071.429780:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488851584:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575071.429784:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488851712:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575071.429796:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488851840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575071.429802:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575071.429817:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575071/real 1788575071] req@ffff94d408c56300 x1875450488851712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575071.429831:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488851712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575071.429837:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575071.429839:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575071.429841:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488851712:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575071.429863:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488851840:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575071.429877:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575071.429886:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575071/real 1788575071] req@ffff94d408c55f80 x1875450488851840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575071.429894:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488851840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575071.429899:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575071.429901:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575071.429903:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488851840:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575071.941312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575072.389368:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575072.965355:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575073.413406:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575073.989356:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575074.437407:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575075.013380:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575075.461401:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575076.037332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575076.485406:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575076.549335:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575076.549344:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575076.549348:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575076.549357:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6224 00000100:00080000:1.0:1788575076.549366:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575076.549373:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575076.549406:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488851968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575076.549428:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575076.549431:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575076.549432:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575076.549435:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6224 00000100:00080000:1.0:1788575076.549437:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575076.549439:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575076.549446:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575076.549455:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488852096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575076.549464:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575076.549466:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575076.549468:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575076.549472:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6224 00000100:00100000:0.0:1788575076.549474:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488851968:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788575076.549475:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575076.549481:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575076.549498:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575076.549506:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488852224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575076.549518:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575076.549520:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575076.549522:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575076.549524:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6224 00000100:00080000:1.0:1788575076.549527:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575076.549530:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575076.549533:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575076.549540:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488852352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575076.549547:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575076.549549:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575076.549551:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575076.549552:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6224 00000100:00080000:1.0:1788575076.549555:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575076.549557:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:0.0:1788575076.549561:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:1.0:1788575076.549562:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575076.549573:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488852480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575076.549597:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575076/real 1788575076] req@ffff94d4049bfb80 x1875450488851968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575076.549610:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488851968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575076.549617:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575076.549620:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575076.549623:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488851968:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575076.549641:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488852096:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575076.549663:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575076.549680:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575076/real 1788575076] req@ffff94d4049bc000 x1875450488852096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575076.549688:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488852096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575076.549693:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575076.549696:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575076.549698:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488852096:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575076.549702:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488852224:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575076.549715:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575076.549721:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575076/real 1788575076] req@ffff94d4049bc700 x1875450488852224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575076.549729:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488852224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575076.549734:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575076.549736:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575076.549738:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488852224:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575076.549741:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488852352:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575076.549751:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575076.549757:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575076/real 1788575076] req@ffff94d4049bed80 x1875450488852352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575076.549764:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488852352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575076.549769:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575076.549771:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575076.549772:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488852352:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575076.549776:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488852480:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575076.549785:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575076.549790:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575076/real 1788575076] req@ffff94d4049bf800 x1875450488852480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575076.549798:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488852480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575076.549803:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575076.549805:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575076.549807:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488852480:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575077.061314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575077.509392:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575078.085378:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575078.533373:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575079.109336:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575079.557338:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575080.133322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575080.581425:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575081.157361:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575081.605403:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575081.669302:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575081.669310:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575081.669313:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575081.669321:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6229 00000100:00080000:1.0:1788575081.669331:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575081.669337:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575081.669374:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488852608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575081.669395:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575081.669398:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575081.669399:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575081.669402:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6229 00000100:00080000:1.0:1788575081.669404:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575081.669411:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575081.669418:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575081.669426:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488852736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575081.669433:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575081.669435:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575081.669437:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575081.669439:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6229 00000100:00080000:1.0:1788575081.669441:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575081.669443:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575081.669447:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575081.669454:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488852864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575081.669461:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575081.669464:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575081.669466:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575081.669469:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6229 00000100:00080000:1.0:1788575081.669471:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575081.669473:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575081.669476:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575081.669486:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488852608:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575081.669488:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488852992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575081.669502:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575081.669503:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575081.669504:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575081.669507:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6229 00000100:00080000:1.0:1788575081.669509:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575081.669511:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575081.669514:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575081.669521:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488853120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575081.669637:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488852736:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575081.669676:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488852864:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575081.669718:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488852992:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575081.669763:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488853120:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575082.181333:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575082.629416:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575083.205336:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575083.653401:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575084.229323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575084.677400:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788575084.741467:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:2.0:1788575084.741504:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788575084.741517:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788575084.741558:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788575084.741570:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788575084.741578:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788575084.741588:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:0.0:1788575084.741656:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575081/real 1788575084] req@ffff94d4049bd500 x1875450488852992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575084.741698:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488852992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575084.741712:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575084.741718:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575084.741722:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488852992:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575084.741731:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575081/real 1788575084] req@ffff94d4049bdc00 x1875450488853120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575084.741749:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488853120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575084.741756:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575084.741758:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575084.741761:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488853120:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575084.741769:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575081/real 1788575084] req@ffff94d4049be680 x1875450488852608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575084.741780:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488852608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575084.741787:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575084.741790:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575084.741793:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488852608:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575084.741800:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575081/real 1788575084] req@ffff94d4049bf100 x1875450488852736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575084.741810:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488852736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575084.741816:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575084.741819:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575084.741821:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488852736:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575084.741828:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575081/real 1788575084] req@ffff94d4049be300 x1875450488852864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575084.741838:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488852864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575084.741846:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575084.741848:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575084.741851:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488852864:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575085.253339:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575085.701392:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575086.277339:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575086.725360:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575086.789387:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575086.789408:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575086.789413:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575086.789426:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6235 00000100:00080000:0.0:1788575086.789441:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575086.789451:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575086.789498:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488853248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575086.789534:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575086.789538:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575086.789540:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575086.789545:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6235 00000100:00080000:0.0:1788575086.789550:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575086.789555:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575086.789567:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575086.789576:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488853248:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575086.789585:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488853376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575086.789618:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575086.789622:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575086.789624:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575086.789628:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6235 00000100:00080000:0.0:1788575086.789645:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575086.789650:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575086.789666:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575086.789679:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488853504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575086.789685:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788575086.789694:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575086.789698:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575086.789700:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575086.789705:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6235 00000100:00080000:0.0:1788575086.789710:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575086.789716:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575086.789722:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575086.789734:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488853632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575086.789739:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575086/real 1788575086] req@ffff94d408d3f480 x1875450488853248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575086.789754:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575086.789759:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575086.789762:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575086.789767:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6235 00000100:00100000:1.0:1788575086.789770:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488853248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575086.789773:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575086.789781:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575086.789788:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788575086.789789:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575086.789798:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575086.789804:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488853248:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575086.789806:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488853760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575086.789841:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488853376:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575086.789872:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575086.789886:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575086/real 1788575086] req@ffff94d408d3c700 x1875450488853376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575086.789898:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488853376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575086.789905:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575086.789909:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575086.789912:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488853376:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575086.789918:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488853504:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575086.789938:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575086.789950:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575086/real 1788575086] req@ffff94d408d3e300 x1875450488853504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575086.789963:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488853504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575086.789970:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575086.789973:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575086.789975:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488853504:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575086.789981:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488853632:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575086.790002:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575086.790012:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575086/real 1788575086] req@ffff94d408d3dc00 x1875450488853632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575086.790023:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488853632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575086.790030:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575086.790033:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575086.790036:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488853632:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575086.790041:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488853760:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575086.790060:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575086.790070:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575086/real 1788575086] req@ffff94d408d3c380 x1875450488853760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575086.790082:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488853760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575086.790095:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575086.790098:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575086.790101:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488853760:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575087.301369:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575087.749366:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575088.325288:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575088.773353:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575089.349319:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575089.797510:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575090.373893:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575090.821416:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575091.398982:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575091.845365:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575091.909356:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575091.909368:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575091.909373:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575091.909386:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6240 00000100:00080000:0.0:1788575091.909400:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575091.909409:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575091.909458:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488853888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575091.909490:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575091.909494:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575091.909496:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575091.909500:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6240 00000100:00080000:0.0:1788575091.909504:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575091.909509:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575091.909520:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575091.909533:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ed80 x1875450488854016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575091.909544:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575091.909546:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488853888:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788575091.909556:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575091.909559:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575091.909564:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6240 00000100:00080000:0.0:1788575091.909567:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575091.909571:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575091.909577:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575091.909590:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488854144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575091.909600:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575091.909603:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575091.909605:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575091.909609:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6240 00000100:00080000:0.0:1788575091.909612:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575091.909616:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575091.909620:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788575091.909627:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575091.909633:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488854272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575091.909654:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575091.909657:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575091.909659:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575091.909663:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6240 00000100:00080000:0.0:1788575091.909668:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575091.909669:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575091/real 1788575091] req@ffff94d408d3d880 x1875450488853888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575091.909675:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575091.909684:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575091.909699:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488854400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575091.909700:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488853888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575091.909725:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575091.909729:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575091.909732:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488853888:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575091.909770:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488854016:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575091.909805:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575091.909818:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575091/real 1788575091] req@ffff94d408d3ed80 x1875450488854016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575091.909831:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ed80 x1875450488854016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575091.909840:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575091.909842:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575091.909845:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488854016:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575091.909856:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488854144:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575091.909874:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575091.909882:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575091/real 1788575091] req@ffff94d408d3c380 x1875450488854144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575091.909889:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488854144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575091.909894:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575091.909896:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575091.909898:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488854144:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575091.909902:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488854272:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575091.909912:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575091.909917:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575091/real 1788575091] req@ffff94d408d3dc00 x1875450488854272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575091.909924:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488854272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575091.909929:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575091.909932:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575091.909936:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488854272:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575091.909941:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488854400:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575091.909958:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575091.909969:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575091/real 1788575091] req@ffff94d408d3e300 x1875450488854400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575091.909980:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488854400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575091.909986:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575091.909987:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575091.909989:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488854400:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575092.421510:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575092.869414:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575093.445657:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575093.893448:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575094.469644:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575094.917451:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575095.493308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575095.941457:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575096.517306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575096.965460:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575097.029283:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575097.029292:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575097.029296:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575097.029305:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6245 00000100:00080000:0.0:1788575097.029314:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575097.029320:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575097.029354:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488854528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575097.029379:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575097.029382:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575097.029384:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575097.029387:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6245 00000100:00080000:0.0:1788575097.029390:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575097.029393:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575097.029401:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575097.029411:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488854656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575097.029421:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575097.029423:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575097.029425:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575097.029431:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6245 00000100:00100000:1.0:1788575097.029431:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488854528:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788575097.029437:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575097.029442:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575097.029446:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575097.029455:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488854784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575097.029462:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575097.029465:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575097.029467:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575097.029472:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6245 00000100:00080000:0.0:1788575097.029482:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575097.029487:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575097.029492:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788575097.029497:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575097.029505:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488854912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575097.029514:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575097.029517:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575097.029519:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575097.029523:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6245 00000100:00080000:0.0:1788575097.029526:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575097.029529:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575097.029533:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575097.029534:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575097/real 1788575097] req@ffff94d408d3c700 x1875450488854528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575097.029543:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488855040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575097.029558:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488854528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575097.029570:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575097.029574:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575097.029579:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488854528:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575097.029600:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488854656:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575097.029621:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575097.029632:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575097/real 1788575097] req@ffff94d408d3f480 x1875450488854656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575097.029643:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488854656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575097.029650:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575097.029652:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575097.029655:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488854656:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575097.029661:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488854784:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575097.029678:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575097.029687:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575097/real 1788575097] req@ffff94d408d3c000 x1875450488854784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575097.029698:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488854784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575097.029706:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575097.029709:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575097.029712:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488854784:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575097.029717:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488854912:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575097.029734:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575097.029745:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575097/real 1788575097] req@ffff94d408d3d180 x1875450488854912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575097.029755:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488854912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575097.029762:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575097.029764:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575097.029767:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488854912:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575097.029772:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488855040:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575097.029789:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575097.029799:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575097/real 1788575097] req@ffff94d408d3df80 x1875450488855040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575097.029808:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488855040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575097.029815:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575097.029818:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575097.029821:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488855040:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575097.541288:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575097.989458:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575098.565254:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575099.013419:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575099.589272:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575100.037441:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575100.613312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575101.061474:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575101.637363:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575102.085427:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575102.149333:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575102.149345:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575102.149350:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575102.149362:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6250 00000100:00080000:2.0:1788575102.149375:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575102.149384:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575102.149428:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60700 x1875450488855168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575102.149459:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575102.149463:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575102.149464:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575102.149468:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6250 00000100:00080000:2.0:1788575102.149471:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575102.149475:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575102.149485:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575102.149497:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60380 x1875450488855296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575102.149507:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575102.149511:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575102.149514:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575102.149519:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488855168:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788575102.149522:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6250 00000100:00080000:2.0:1788575102.149531:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575102.149538:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575102.149543:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575102.149555:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62300 x1875450488855424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575102.149563:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575102.149567:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575102.149569:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575102.149573:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6250 00000100:00080000:2.0:1788575102.149580:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575102.149584:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575102.149590:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575102.149603:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488855552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575102.149610:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788575102.149626:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575102.149630:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575102.149632:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575102.149640:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6250 00000100:00080000:2.0:1788575102.149644:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575102.149647:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575102.149651:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575102.149661:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575102/real 1788575102] req@ffff94d408c60700 x1875450488855168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575102.149664:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488855680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575102.149738:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60700 x1875450488855168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575102.149751:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575102.149757:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575102.149761:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488855168:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575102.149784:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488855296:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575102.149803:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575102.149812:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575102/real 1788575102] req@ffff94d408c60380 x1875450488855296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575102.149820:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60380 x1875450488855296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575102.149825:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575102.149827:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575102.149829:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488855296:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575102.149833:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488855424:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575102.149843:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575102.149848:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575102/real 1788575102] req@ffff94d408c62300 x1875450488855424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575102.149854:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62300 x1875450488855424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575102.149863:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575102.149865:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575102.149867:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488855424:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575102.149870:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488855552:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575102.149879:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575102.149884:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575102/real 1788575102] req@ffff94d408c62680 x1875450488855552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575102.149890:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488855552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575102.149894:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575102.149896:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575102.149898:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488855552:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575102.149901:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488855680:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575102.149909:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575102.149937:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575102/real 1788575102] req@ffff94d408c63800 x1875450488855680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575102.149943:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488855680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575102.149948:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575102.149950:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575102.149953:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488855680:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575102.661282:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575103.109429:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575103.685335:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575104.133382:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575104.709281:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575105.157446:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575105.733314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575106.181364:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575106.757330:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575107.205423:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575107.269342:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575107.269354:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575107.269359:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575107.269371:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6255 00000100:00080000:2.0:1788575107.269383:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575107.269392:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575107.269432:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488855808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575107.269460:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575107.269464:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575107.269466:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575107.269470:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6255 00000100:00080000:2.0:1788575107.269472:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575107.269476:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575107.269487:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575107.269501:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488855936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575107.269512:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488855808:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788575107.269512:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575107.269518:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575107.269522:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575107.269526:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6255 00000100:00080000:2.0:1788575107.269529:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575107.269533:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575107.269537:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575107.269549:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488856064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575107.269557:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575107.269560:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575107.269562:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575107.269566:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6255 00000100:00080000:2.0:1788575107.269569:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575107.269573:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575107.269579:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575107.269591:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488856192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575107.269594:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788575107.269604:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575107.269609:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575107.269612:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575107.269617:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6255 00000100:00080000:2.0:1788575107.269622:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575107.269627:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575107.269632:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575107.269643:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575107/real 1788575107] req@ffff94d408c60000 x1875450488855808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575107.269646:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488856320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575107.269682:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488855808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575107.269696:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575107.269700:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575107.269705:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488855808:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575107.269734:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488855936:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575107.269764:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575107.269779:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575107/real 1788575107] req@ffff94d408c62d80 x1875450488855936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575107.269792:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488855936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575107.269800:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575107.269803:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575107.269807:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488855936:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575107.269813:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488856064:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575107.269833:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575107.269845:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575107/real 1788575107] req@ffff94d408c61180 x1875450488856064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575107.269858:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488856064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575107.269867:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575107.269871:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575107.269875:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488856064:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575107.269884:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488856192:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575107.269916:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575107.269929:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575107/real 1788575107] req@ffff94d408c62a00 x1875450488856192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575107.269943:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488856192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575107.269950:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575107.269954:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575107.269958:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488856192:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575107.269964:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488856320:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575107.269982:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575107.269993:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575107/real 1788575107] req@ffff94d408c63b80 x1875450488856320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575107.270032:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488856320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575107.270040:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575107.270043:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575107.270046:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488856320:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575107.781311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575108.229440:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575108.805302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575109.253401:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575109.829281:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575110.277412:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575110.853314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575111.301348:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575111.877487:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575112.325411:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575112.389360:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575112.389370:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575112.389374:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575112.389385:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6260 00000100:00080000:1.0:1788575112.389396:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575112.389403:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575112.389442:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd880 x1875450488856448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575112.389471:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575112.389473:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575112.389475:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575112.389478:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6260 00000100:00080000:1.0:1788575112.389480:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575112.389482:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575112.389490:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575112.389499:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488856576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575112.389508:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575112.389512:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575112.389513:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575112.389518:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6260 00000100:00080000:1.0:1788575112.389521:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575112.389524:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575112.389531:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488856448:192.168.202.125@tcp:250:kworker.0 00000008:00100000:1.0:1788575112.389531:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575112.389548:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488856704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575112.389569:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575112.389573:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575112.389575:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575112.389579:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6260 00000100:00080000:1.0:1788575112.389594:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575112.389599:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575112.389605:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788575112.389612:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575112.389619:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488856832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575112.389628:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575112.389641:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575112.389643:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575112.389647:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6260 00000100:00080000:1.0:1788575112.389651:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575112.389655:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575112.389661:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575112.389664:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575112/real 1788575112] req@ffff94d4049bd880 x1875450488856448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575112.389675:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488856960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575112.389687:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd880 x1875450488856448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575112.389700:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575112.389705:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575112.389709:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488856448:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575112.389717:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488856576:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575112.389737:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575112.389747:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575112/real 1788575112] req@ffff94d4049bca80 x1875450488856576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575112.389763:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488856576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575112.389771:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575112.389774:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575112.389777:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488856576:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575112.389808:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488856704:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575112.389825:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575112.389835:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575112/real 1788575112] req@ffff94d4049be300 x1875450488856704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575112.389845:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488856704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575112.389852:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575112.389855:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575112.389858:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488856704:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575112.389863:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488856832:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575112.389879:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575112.389887:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575112/real 1788575112] req@ffff94d4049bf100 x1875450488856832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575112.389897:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488856832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575112.389904:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575112.389907:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575112.389910:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488856832:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575112.389915:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488856960:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575112.389930:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575112.389940:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575112/real 1788575112] req@ffff94d4049be680 x1875450488856960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575112.389950:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488856960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575112.389957:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575112.389959:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575112.389962:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488856960:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575112.901289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575113.349435:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575113.925371:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575114.373427:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575114.949352:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575115.397427:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575115.973353:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575116.421387:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575116.997335:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575117.445436:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575117.509334:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575117.509347:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575117.509352:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575117.509369:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6265 00000100:00080000:2.0:1788575117.509384:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575117.509392:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575117.509440:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488857088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575117.509471:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575117.509475:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575117.509477:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575117.509480:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6265 00000100:00080000:2.0:1788575117.509484:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575117.509488:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575117.509498:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575117.509512:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488857216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575117.509516:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488857088:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788575117.509538:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575117.509541:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575117.509543:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575117.509548:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6265 00000100:00080000:2.0:1788575117.509552:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575117.509557:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575117.509562:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575117.509575:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488857344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575117.509584:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575117.509588:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1788575117.509592:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788575117.509593:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575117.509599:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6265 00000100:00080000:2.0:1788575117.509605:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575117.509609:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575117.509616:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575117.509637:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575117/real 1788575117] req@ffff94d408c60a80 x1875450488857088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575117.509637:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488857472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575117.509666:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575117.509670:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575117.509672:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575117.509678:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488857088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575117.509678:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6265 00000100:00080000:2.0:1788575117.509684:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575117.509693:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00080000:0.0:1788575117.509698:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1788575117.509699:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788575117.509707:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575117.509712:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488857088:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575117.509716:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488857600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575117.509740:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488857216:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575117.509760:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575117.509769:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575117/real 1788575117] req@ffff94d408c63480 x1875450488857216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575117.509782:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488857216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575117.509788:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575117.509790:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575117.509792:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488857216:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575117.509805:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488857344:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575117.509819:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575117.509829:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575117/real 1788575117] req@ffff94d408c61f80 x1875450488857344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575117.509841:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488857344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575117.509849:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575117.509852:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575117.509856:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488857344:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575117.509862:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488857472:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575117.509879:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575117.509887:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575117/real 1788575117] req@ffff94d408c61500 x1875450488857472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575117.509894:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488857472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575117.509904:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575117.509906:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575117.509908:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488857472:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575117.509911:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488857600:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575117.509921:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575117.509926:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575117/real 1788575117] req@ffff94d408c61880 x1875450488857600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575117.509933:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488857600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575117.509937:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575117.509939:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575117.509940:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488857600:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575118.021345:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575118.469407:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575119.045354:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575119.493398:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575120.069317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575120.517433:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575121.093298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575121.541344:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575122.117292:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575122.565371:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575122.629337:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575122.629345:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575122.629348:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575122.629355:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6270 00000100:00080000:0.0:1788575122.629363:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575122.629371:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575122.629405:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488857728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575122.629439:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575122.629443:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575122.629445:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575122.629449:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6270 00000100:00080000:0.0:1788575122.629453:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575122.629457:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575122.629466:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575122.629479:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488857728:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575122.629480:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488857856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575122.629493:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575122.629497:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575122.629499:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575122.629503:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6270 00000100:00080000:0.0:1788575122.629507:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575122.629511:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575122.629515:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575122.629527:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488857984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575122.629534:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788575122.629541:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575122.629545:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575122.629548:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575122.629553:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6270 00000100:00080000:0.0:1788575122.629557:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575122.629561:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575122.629566:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575122.629572:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575122/real 1788575122] req@ffff94d408d3ce00 x1875450488857728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575122.629581:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488858112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575122.629593:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575122.629598:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1788575122.629598:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488857728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575122.629601:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575122.629609:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575122.629615:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:0.0:1788575122.629618:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6270 00000100:00100000:1.0:1788575122.629621:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488857728:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788575122.629624:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575122.629631:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575122.629635:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575122.629643:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488857856:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575122.629648:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488858240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575122.629663:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575122.629677:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575122/real 1788575122] req@ffff94d408d3d500 x1875450488857856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575122.629691:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488857856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575122.629696:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575122.629698:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575122.629700:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488857856:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575122.629703:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488857984:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575122.629713:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575122.629718:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575122/real 1788575122] req@ffff94d408d3ca80 x1875450488857984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575122.629724:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488857984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575122.629729:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575122.629730:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575122.629732:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488857984:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575122.629735:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488858112:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575122.629744:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575122.629749:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575122/real 1788575122] req@ffff94d408d3ea00 x1875450488858112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575122.629755:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488858112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575122.629763:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575122.629765:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575122.629766:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488858112:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575122.629774:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488858240:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575122.629783:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575122.629787:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575122/real 1788575122] req@ffff94d408d3e680 x1875450488858240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575122.629793:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488858240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575122.629798:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575122.629800:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575122.629801:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488858240:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575123.141316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575123.589379:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575124.165331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575124.613315:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575125.189325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575125.637402:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575126.213306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575126.661394:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575127.237264:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575127.685366:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575127.749323:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575127.749332:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575127.749337:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575127.749358:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6276 00000100:00080000:2.0:1788575127.749368:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575127.749376:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575127.749409:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488858368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575127.749434:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575127.749438:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575127.749440:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575127.749444:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6276 00000100:00080000:2.0:1788575127.749447:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575127.749453:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575127.749461:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575127.749469:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488858368:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575127.749476:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60e00 x1875450488858496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575127.749488:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575127.749494:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575127.749496:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575127.749507:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6276 00000100:00080000:2.0:1788575127.749511:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575127.749516:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575127.749521:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575127.749528:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575127.749534:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63100 x1875450488858624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575127.749545:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575127.749549:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575127.749551:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575127.749555:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6276 00000100:00080000:2.0:1788575127.749560:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575127.749564:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575127.749568:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575127/real 1788575127] req@ffff94d408c61c00 x1875450488858368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788575127.749571:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575127.749588:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488858752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575127.749592:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488858368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575127.749611:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575127.749616:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575127.749617:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575127.749620:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575127.749622:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575127.749628:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488858368:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788575127.749628:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6276 00000100:00080000:2.0:1788575127.749642:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575127.749648:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575127.749651:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488858496:192.168.202.125@tcp:38:kworker.0 00000008:00100000:2.0:1788575127.749655:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575127.749670:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488858880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575127.749673:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575127.749685:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575127/real 1788575127] req@ffff94d408c60e00 x1875450488858496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575127.749694:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60e00 x1875450488858496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575127.749699:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575127.749701:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575127.749704:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488858496:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575127.749708:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488858624:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575127.749718:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575127.749723:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575127/real 1788575127] req@ffff94d408c63100 x1875450488858624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575127.749729:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63100 x1875450488858624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575127.749734:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575127.749736:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575127.749738:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488858624:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575127.749741:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488858752:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575127.749758:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575127.749765:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575127/real 1788575127] req@ffff94d408c61880 x1875450488858752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575127.749771:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488858752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575127.749776:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575127.749778:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575127.749779:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488858752:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575127.749788:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488858880:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575127.749798:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575127.749802:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575127/real 1788575127] req@ffff94d408c61500 x1875450488858880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575127.749809:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488858880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575127.749813:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575127.749815:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575127.749816:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488858880:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575128.261381:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575128.709371:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575129.285375:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575129.733406:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575130.309319:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575130.757419:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575131.333259:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575131.781419:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575132.357245:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575132.805322:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575132.869376:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575132.869389:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575132.869394:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575132.869403:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6281 00000100:00080000:0.0:1788575132.869415:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575132.869423:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575132.869484:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f100 x1875450488859008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575132.869510:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575132.869513:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575132.869518:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575132.869521:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6281 00000100:00080000:0.0:1788575132.869523:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575132.869526:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575132.869536:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575132.869544:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f800 x1875450488859136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575132.869551:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575132.869554:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575132.869555:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575132.869559:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6281 00000100:00080000:0.0:1788575132.869562:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575132.869563:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488859008:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788575132.869564:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575132.869570:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575132.869578:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488859264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575132.869589:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575132.869591:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575132.869592:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575132.869595:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6281 00000100:00080000:0.0:1788575132.869598:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575132.869601:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575132.869604:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575132.869611:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488859392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575132.869617:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575132.869619:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575132.869620:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575132.869622:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6281 00000100:00080000:0.0:1788575132.869637:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575132.869640:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575132.869642:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575132.869649:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488859520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575132.869653:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575132.869695:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575132/real 1788575132] req@ffff94d408d3f100 x1875450488859008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575132.869709:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f100 x1875450488859008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575132.869717:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575132.869720:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575132.869723:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488859008:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575132.869746:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488859136:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575132.869761:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575132.869769:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575132/real 1788575132] req@ffff94d408d3f800 x1875450488859136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575132.869776:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f800 x1875450488859136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575132.869782:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575132.869784:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575132.869786:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488859136:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575132.869789:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488859264:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575132.869797:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575132.869802:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575132/real 1788575132] req@ffff94d408d3e680 x1875450488859264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575132.869813:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488859264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575132.869818:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575132.869819:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575132.869821:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488859264:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575132.869823:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488859392:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575132.869833:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575132.869838:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575132/real 1788575132] req@ffff94d408d3ea00 x1875450488859392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575132.869845:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488859392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575132.869849:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575132.869850:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575132.869852:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488859392:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575132.869855:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488859520:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575132.869863:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575132.869868:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575132/real 1788575132] req@ffff94d408d3ca80 x1875450488859520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575132.869873:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488859520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575132.869881:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575132.869883:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575132.869885:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488859520:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575133.381266:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575133.829438:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575134.405309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575134.853393:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575135.429272:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575135.877410:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575136.453368:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575136.901434:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575137.477337:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575137.925425:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575137.989280:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575137.989291:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575137.989295:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575137.989304:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6286 00000100:00080000:1.0:1788575137.989315:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575137.989323:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575137.989358:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488859648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575137.989379:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575137.989382:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575137.989383:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575137.989385:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6286 00000100:00080000:1.0:1788575137.989388:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575137.989390:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575137.989398:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575137.989406:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488859776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575137.989417:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575137.989420:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788575137.989421:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488859648:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788575137.989421:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575137.989427:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6286 00000100:00080000:1.0:1788575137.989432:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575137.989434:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575137.989436:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575137.989456:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488859904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575137.989462:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575137.989464:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575137.989465:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575137.989468:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6286 00000100:00080000:1.0:1788575137.989470:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575137.989473:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575137.989477:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575137.989483:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488860032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575137.989488:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575137.989491:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575137.989492:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575137.989494:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6286 00000100:00080000:1.0:1788575137.989497:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575137.989499:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575137.989502:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575137.989510:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488860160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575137.989511:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575137.989552:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575137/real 1788575137] req@ffff94d4049bdc00 x1875450488859648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575137.989568:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488859648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575137.989577:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575137.989581:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575137.989585:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488859648:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575137.989605:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488859776:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575137.989637:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575137.989649:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575137/real 1788575137] req@ffff94d4049bd500 x1875450488859776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575137.989659:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488859776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575137.989665:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575137.989668:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575137.989670:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488859776:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575137.989676:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488859904:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575137.989692:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575137.989701:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575137/real 1788575137] req@ffff94d4049bf800 x1875450488859904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575137.989711:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488859904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575137.989717:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575137.989720:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575137.989722:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488859904:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575137.989728:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488860032:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575137.989746:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575137.989756:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575137/real 1788575137] req@ffff94d4049bed80 x1875450488860032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575137.989766:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488860032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575137.989773:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575137.989776:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575137.989778:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488860032:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575137.989784:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488860160:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575137.989802:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575137.989812:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575137/real 1788575137] req@ffff94d4049bc700 x1875450488860160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575137.989822:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488860160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575137.989830:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575137.989833:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575137.989836:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488860160:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575138.501289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575138.949345:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575139.525319:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575139.973371:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575140.549312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575140.997371:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575141.573312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575142.021366:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575142.597326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575143.045386:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575143.109330:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575143.109340:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575143.109345:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575143.109358:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6291 00000100:00080000:1.0:1788575143.109374:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575143.109385:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575143.109434:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488860288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575143.109464:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575143.109467:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575143.109469:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575143.109473:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6291 00000100:00080000:1.0:1788575143.109478:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575143.109481:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575143.109491:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575143.109506:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488860416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575143.109512:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488860288:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788575143.109519:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575143.109525:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575143.109526:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575143.109532:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6291 00000100:00080000:1.0:1788575143.109536:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575143.109539:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575143.109544:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575143.109559:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488860544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575143.109566:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575143.109570:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575143.109572:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575143.109575:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6291 00000100:00080000:1.0:1788575143.109578:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575143.109581:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575143.109585:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575143.109594:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488860672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788575143.109605:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788575143.109609:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575143.109612:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575143.109615:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575143.109618:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6291 00000100:00080000:1.0:1788575143.109624:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575143.109629:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575143.109633:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575143.109647:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488860800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575143.109669:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575143/real 1788575143] req@ffff94d4049bc000 x1875450488860288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575143.109690:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488860288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575143.109700:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575143.109704:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575143.109708:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488860288:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575143.109735:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488860416:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575143.109760:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575143.109772:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575143/real 1788575143] req@ffff94d4049bfb80 x1875450488860416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575143.109782:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488860416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575143.109790:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575143.109793:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575143.109797:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488860416:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575143.109803:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488860544:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575143.109823:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575143.109835:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575143/real 1788575143] req@ffff94d4049bd180 x1875450488860544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575143.109857:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488860544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575143.109867:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575143.109870:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575143.109873:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488860544:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575143.109880:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488860672:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575143.109903:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575143.109916:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575143/real 1788575143] req@ffff94d4049bce00 x1875450488860672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575143.109930:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488860672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575143.109938:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575143.109942:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575143.109945:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488860672:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575143.109952:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488860800:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575143.109973:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575143.109993:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575143/real 1788575143] req@ffff94d4049bea00 x1875450488860800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575143.110021:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488860800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575143.110032:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575143.110036:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575143.110041:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488860800:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575143.621318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575144.069358:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575144.645293:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575145.093360:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575145.669312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575146.117348:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575146.693270:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575147.141315:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575147.717323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575148.165336:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575148.229318:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575148.229325:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575148.229328:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575148.229336:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6296 00000100:00080000:2.0:1788575148.229344:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575148.229349:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575148.229378:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488860928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575148.229402:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575148.229405:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575148.229406:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575148.229408:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6296 00000100:00080000:2.0:1788575148.229410:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575148.229412:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575148.229419:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575148.229426:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488861056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575148.229438:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575148.229440:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575148.229441:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575148.229443:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6296 00000100:00080000:2.0:1788575148.229445:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575148.229448:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575148.229451:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575148.229457:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488861184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575148.229467:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488860928:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788575148.229468:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575148.229475:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575148.229479:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575148.229485:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6296 00000100:00080000:2.0:1788575148.229487:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575148.229489:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575148.229491:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575148.229499:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488861312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575148.229510:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575148.229512:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575148.229513:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575148.229516:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6296 00000100:00080000:2.0:1788575148.229519:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575148.229521:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575148.229524:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575148.229530:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488861440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575148.229600:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488861056:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575148.229645:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488861184:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575148.229664:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488861312:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575148.229678:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488861440:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575148.741321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575149.189395:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575149.765323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575150.213327:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575150.789341:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575151.237381:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788575151.301433:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:1.0:1788575151.301468:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788575151.301483:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575151.301517:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575151.301528:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575151.301537:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575151.301545:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:0.0:1788575151.301562:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575148/real 1788575151] req@ffff94d408c60a80 x1875450488861184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575151.301596:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488861184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575151.301610:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575151.301616:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575151.301620:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488861184:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575151.301628:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575148/real 1788575151] req@ffff94d408c63b80 x1875450488861312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575151.301659:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488861312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575151.301668:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575151.301672:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575151.301676:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488861312:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575151.301684:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575148/real 1788575151] req@ffff94d408c62a00 x1875450488861440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575151.301705:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488861440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575151.301713:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575151.301717:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575151.301720:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488861440:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575151.301727:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575148/real 1788575151] req@ffff94d408c61f80 x1875450488860928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575151.301739:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488860928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575151.301746:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575151.301749:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575151.301753:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488860928:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575151.301760:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575148/real 1788575151] req@ffff94d408c63480 x1875450488861056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575151.301771:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488861056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575151.301779:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575151.301783:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575151.301786:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488861056:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575151.813269:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575152.261350:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575152.837305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575153.285361:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575153.349337:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575153.349345:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575153.349348:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575153.349356:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6301 00000100:00080000:0.0:1788575153.349365:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575153.349372:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575153.349404:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488861568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575153.349433:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575153.349438:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575153.349439:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575153.349443:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6301 00000100:00080000:0.0:1788575153.349447:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575153.349452:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575153.349463:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575153.349480:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488861568:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575153.349481:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488861696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575153.349500:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575153.349504:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575153.349506:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575153.349531:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6301 00000100:00080000:0.0:1788575153.349536:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575153.349541:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575153.349547:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575153.349562:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488861824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575153.349574:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788575153.349576:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788575153.349582:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575153.349585:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575153.349590:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6301 00000100:00080000:0.0:1788575153.349595:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575153.349601:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575153.349607:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575153.349621:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488861952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575153.349623:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575153/real 1788575153] req@ffff94d408d3d500 x1875450488861568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575153.349653:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575153.349657:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1788575153.349660:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488861568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575153.349661:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575153.349672:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6301 00000100:00080000:2.0:1788575153.349677:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575153.349679:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575153.349689:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:0.0:1788575153.349690:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575153.349698:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575153.349703:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488861568:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575153.349714:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488862080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575153.349741:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488861696:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575153.349772:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575153.349786:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575153/real 1788575153] req@ffff94d408d3ce00 x1875450488861696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575153.349798:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488861696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575153.349808:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575153.349811:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575153.349815:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488861696:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575153.349821:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488861824:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575153.349841:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575153.349853:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575153/real 1788575153] req@ffff94d408d3df80 x1875450488861824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575153.349863:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488861824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575153.349871:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575153.349874:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575153.349877:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488861824:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575153.349883:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488861952:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575153.349903:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575153.349915:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575153/real 1788575153] req@ffff94d408d3d180 x1875450488861952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575153.349926:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488861952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575153.349933:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575153.349936:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575153.349940:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488861952:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575153.349946:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488862080:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575153.349966:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575153.349977:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575153/real 1788575153] req@ffff94d408d3c000 x1875450488862080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575153.349987:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488862080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575153.349994:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575153.349997:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575153.350001:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488862080:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575153.861306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575154.309350:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575154.885301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575155.333384:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575155.909315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575156.357379:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575156.933429:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575157.381417:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575157.959496:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575158.405388:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575158.469315:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575158.469323:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575158.469327:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575158.469341:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6306 00000100:00080000:1.0:1788575158.469350:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575158.469356:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575158.469388:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488862208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575158.469408:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575158.469411:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575158.469412:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575158.469415:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6306 00000100:00080000:1.0:1788575158.469417:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575158.469420:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575158.469427:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575158.469436:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488862336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575158.469444:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575158.469447:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788575158.469447:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488862208:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788575158.469451:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575158.469458:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6306 00000100:00080000:1.0:1788575158.469460:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575158.469463:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575158.469466:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575158.469476:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488862464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575158.469482:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575158.469485:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575158.469486:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575158.469489:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6306 00000100:00080000:1.0:1788575158.469492:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575158.469494:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575158.469497:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575158.469505:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488862592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788575158.469511:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788575158.469517:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575158.469520:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575158.469522:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575158.469526:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6306 00000100:00080000:1.0:1788575158.469529:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575158.469531:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575158.469535:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575158.469542:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488862720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575158.469548:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575158/real 1788575158] req@ffff94d4049bc380 x1875450488862208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575158.469566:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc380 x1875450488862208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575158.469575:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575158.469579:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575158.469582:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488862208:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575158.469599:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488862336:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575158.469615:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575158.469623:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575158/real 1788575158] req@ffff94d4049bf480 x1875450488862336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575158.469631:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488862336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575158.469641:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575158.469643:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575158.469645:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488862336:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575158.469649:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488862464:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575158.469658:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575158.469664:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575158/real 1788575158] req@ffff94d4049bea00 x1875450488862464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575158.469684:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488862464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575158.469693:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575158.469696:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575158.469700:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488862464:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575158.469706:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488862592:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575158.469728:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575158.469740:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575158/real 1788575158] req@ffff94d4049bce00 x1875450488862592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575158.469752:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488862592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575158.469761:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575158.469764:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575158.469768:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488862592:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575158.469783:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488862720:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575158.469805:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575158.469816:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575158/real 1788575158] req@ffff94d4049bd180 x1875450488862720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575158.469824:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488862720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575158.469830:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575158.469832:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575158.469834:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488862720:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575158.981370:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575159.429380:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575160.005348:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575160.453403:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575161.029329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575161.477501:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575162.054622:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575162.501410:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575163.077312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575163.525371:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575163.589283:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575163.589291:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575163.589295:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575163.589304:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6311 00000100:00080000:1.0:1788575163.589313:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575163.589319:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575163.589349:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488862848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575163.589378:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575163.589381:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575163.589383:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575163.589385:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6311 00000100:00080000:1.0:1788575163.589388:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575163.589390:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575163.589397:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575163.589405:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488862976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575163.589413:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575163.589415:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575163.589417:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575163.589419:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6311 00000100:00080000:1.0:1788575163.589421:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575163.589426:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575163.589427:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488862848:192.168.202.125@tcp:250:kworker.0 00000008:00100000:1.0:1788575163.589429:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575163.589439:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488863104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575163.589446:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575163.589448:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575163.589449:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575163.589452:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6311 00000100:00080000:1.0:1788575163.589456:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575163.589458:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575163.589462:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575163.589470:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488863232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575163.589476:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575163.589478:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575163.589479:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575163.589483:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6311 00000800:00000100:2.0:1788575163.589487:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788575163.589489:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575163.589494:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575163.589497:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575163.589506:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488863360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575163.589524:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575163/real 1788575163] req@ffff94d4049bfb80 x1875450488862848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575163.589538:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488862848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575163.589545:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575163.589548:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575163.589551:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488862848:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575163.589556:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488862976:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575163.589571:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575163.589578:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575163/real 1788575163] req@ffff94d4049bc000 x1875450488862976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575163.589585:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488862976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575163.589591:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575163.589593:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575163.589595:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488862976:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575163.589613:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488863104:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575163.589639:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575163.589650:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575163/real 1788575163] req@ffff94d4049bc700 x1875450488863104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575163.589663:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488863104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575163.589672:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575163.589676:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575163.589680:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488863104:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575163.589687:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488863232:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575163.589708:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575163.589717:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575163/real 1788575163] req@ffff94d4049bed80 x1875450488863232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575163.589729:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488863232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575163.589735:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575163.589737:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575163.589739:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488863232:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575163.589742:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488863360:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575163.589752:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575163.589758:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575163/real 1788575163] req@ffff94d4049bf800 x1875450488863360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575163.589765:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488863360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575163.589770:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575163.589772:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575163.589773:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488863360:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575164.101290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575164.549317:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575165.125345:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575165.573394:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575166.149330:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575166.597424:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575167.173328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575167.621399:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575168.197288:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575168.645340:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575168.709344:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575168.709355:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575168.709360:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575168.709372:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6317 00000100:00080000:1.0:1788575168.709383:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575168.709392:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575168.709429:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488863488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575168.709450:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575168.709453:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575168.709454:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575168.709461:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6317 00000100:00080000:1.0:1788575168.709464:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575168.709467:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575168.709475:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575168.709483:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488863616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575168.709491:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575168.709496:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788575168.709497:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488863488:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788575168.709498:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575168.709506:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6317 00000100:00080000:1.0:1788575168.709508:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575168.709510:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575168.709514:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575168.709523:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488863744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575168.709533:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575168.709537:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575168.709539:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575168.709542:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6317 00000100:00080000:1.0:1788575168.709547:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575168.709551:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575168.709557:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788575168.709558:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575168.709574:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488863872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575168.709597:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575168.709599:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575168.709601:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788575168.709604:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575168/real 1788575168] req@ffff94d4049bd500 x1875450488863488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575168.709606:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6317 00000100:00080000:1.0:1788575168.709613:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575168.709619:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575168.709624:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575168.709630:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488863488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575168.709635:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488864000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575168.709649:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575168.709658:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575168.709662:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488863488:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575168.709681:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488863616:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575168.709698:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575168.709707:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575168/real 1788575168] req@ffff94d4049bdc00 x1875450488863616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575168.709715:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488863616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575168.709720:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575168.709723:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575168.709725:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488863616:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575168.709728:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488863744:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575168.709738:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575168.709744:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575168/real 1788575168] req@ffff94d4049be680 x1875450488863744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575168.709751:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488863744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575168.709761:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575168.709763:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575168.709765:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488863744:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575168.709768:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488863872:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575168.709778:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575168.709783:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575168/real 1788575168] req@ffff94d4049bf100 x1875450488863872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575168.709790:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488863872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575168.709795:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575168.709797:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575168.709799:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488863872:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575168.709802:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488864000:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575168.709811:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575168.709817:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575168/real 1788575168] req@ffff94d4049be300 x1875450488864000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575168.709823:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488864000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575168.709828:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575168.709830:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575168.709832:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488864000:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575169.221273:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575169.669365:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575170.245328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575170.693356:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575171.269297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575171.717332:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575172.293328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575172.741393:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575173.317250:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575173.765364:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575173.829349:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575173.829362:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575173.829367:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575173.829378:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6322 00000100:00080000:0.0:1788575173.829391:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575173.829399:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575173.829443:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488864128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575173.829472:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575173.829476:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575173.829477:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575173.829480:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6322 00000100:00080000:0.0:1788575173.829483:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575173.829486:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575173.829495:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575173.829506:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488864256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575173.829513:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575173.829517:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575173.829519:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575173.829523:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6322 00000100:00080000:0.0:1788575173.829525:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575173.829529:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575173.829533:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575173.829539:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488864128:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575173.829547:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488864384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575173.829556:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575173.829558:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575173.829561:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575173.829566:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6322 00000100:00080000:0.0:1788575173.829569:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575173.829572:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575173.829578:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575173.829588:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488864512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575173.829594:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575173.829597:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575173.829599:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575173.829601:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6322 00000100:00080000:0.0:1788575173.829605:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575173.829614:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:2.0:1788575173.829615:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:0.0:1788575173.829622:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575173.829636:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488864640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575173.829663:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575173/real 1788575173] req@ffff94d408d3f480 x1875450488864128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575173.829685:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488864128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575173.829693:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575173.829697:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575173.829700:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488864128:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575173.829707:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488864256:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575173.829725:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575173.829732:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575173/real 1788575173] req@ffff94d408d3c700 x1875450488864256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575173.829740:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488864256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575173.829746:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575173.829748:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575173.829751:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488864256:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575173.829771:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488864384:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575173.829782:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575173.829788:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575173/real 1788575173] req@ffff94d408d3e300 x1875450488864384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575173.829796:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488864384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575173.829801:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575173.829803:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575173.829805:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488864384:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575173.829809:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488864512:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575173.829820:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575173.829825:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575173/real 1788575173] req@ffff94d408d3dc00 x1875450488864512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575173.829833:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488864512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575173.829837:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575173.829839:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575173.829841:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488864512:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575173.829844:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488864640:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575173.829853:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575173.829859:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575173/real 1788575173] req@ffff94d408d3c380 x1875450488864640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575173.829867:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488864640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575173.829871:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575173.829873:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575173.829875:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488864640:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575174.341358:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575174.789409:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575175.365319:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575175.813434:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575176.389359:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575176.837430:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575177.413367:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575177.861452:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575178.437348:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575178.885400:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575178.949276:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575178.949290:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575178.949295:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575178.949313:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6327 00000100:00080000:0.0:1788575178.949327:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575178.949337:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575178.949379:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ed80 x1875450488864768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575178.949412:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575178.949416:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575178.949419:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575178.949423:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6327 00000100:00080000:0.0:1788575178.949427:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575178.949432:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575178.949442:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575178.949457:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488864896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575178.949468:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575178.949472:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575178.949475:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575178.949480:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6327 00000100:00080000:0.0:1788575178.949484:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575178.949491:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575178.949493:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488864768:192.168.202.125@tcp:250:kworker.0 00000008:00100000:0.0:1788575178.949498:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575178.949514:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488865024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575178.949525:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575178.949529:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575178.949531:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575178.949535:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6327 00000100:00080000:0.0:1788575178.949539:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575178.949543:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575178.949548:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575178.949560:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488865152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788575178.949562:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788575178.949574:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575178.949578:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575178.949580:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575178.949586:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6327 00000100:00080000:0.0:1788575178.949591:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575178.949597:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575178.949600:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575178/real 1788575178] req@ffff94d408d3ed80 x1875450488864768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788575178.949604:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575178.949622:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ed80 x1875450488864768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575178.949624:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488865280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575178.949639:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575178.949644:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575178.949648:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488864768:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575178.949655:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488864896:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575178.949672:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575178.949685:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575178/real 1788575178] req@ffff94d408d3d880 x1875450488864896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575178.949692:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488864896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575178.949697:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575178.949699:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575178.949701:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488864896:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575178.949721:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488865024:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575178.949730:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575178.949735:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575178/real 1788575178] req@ffff94d408d3c380 x1875450488865024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575178.949742:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488865024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575178.949747:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575178.949748:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575178.949750:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488865024:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575178.949765:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488865152:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575178.949778:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575178.949784:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575178/real 1788575178] req@ffff94d408d3dc00 x1875450488865152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575178.949794:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488865152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575178.949802:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575178.949805:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575178.949808:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488865152:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575178.949814:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488865280:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575178.949834:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575178.949843:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575178/real 1788575178] req@ffff94d408d3e300 x1875450488865280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575178.949850:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488865280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575178.949855:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575178.949857:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575178.949858:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488865280:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575179.461329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575179.909394:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575180.485309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575180.933350:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575181.509308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575181.957374:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575182.533290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575182.981325:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575183.557260:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575184.005335:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575184.069274:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575184.069281:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575184.069284:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575184.069292:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6332 00000100:00080000:0.0:1788575184.069300:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575184.069306:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575184.069335:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488865408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575184.069359:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575184.069363:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575184.069364:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575184.069368:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6332 00000100:00080000:0.0:1788575184.069371:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575184.069375:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575184.069384:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575184.069396:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488865408:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575184.069401:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488865536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575184.069413:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575184.069417:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575184.069419:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575184.069424:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6332 00000100:00080000:0.0:1788575184.069427:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575184.069431:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575184.069437:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788575184.069449:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575184.069450:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488865664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575184.069458:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575184.069461:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575184.069462:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575184.069466:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6332 00000100:00080000:0.0:1788575184.069469:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575184.069472:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575184.069475:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575184.069479:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575184/real 1788575184] req@ffff94d408d3c700 x1875450488865408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575184.069483:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488865792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575184.069492:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575184.069494:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575184.069496:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575184.069502:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6332 00000100:00080000:0.0:1788575184.069505:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575184.069507:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575184.069510:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575184.069511:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488865408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575184.069518:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488865920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575184.069521:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575184.069528:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575184.069533:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488865408:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575184.069552:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488865536:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575184.069573:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575184.069584:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575184/real 1788575184] req@ffff94d408d3f480 x1875450488865536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575184.069595:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488865536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575184.069603:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575184.069606:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575184.069609:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488865536:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575184.069615:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488865664:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575184.069631:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575184.069641:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575184/real 1788575184] req@ffff94d408d3c000 x1875450488865664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575184.069679:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488865664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575184.069689:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575184.069692:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575184.069696:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488865664:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575184.069702:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488865792:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575184.069720:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575184.069729:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575184/real 1788575184] req@ffff94d408d3d180 x1875450488865792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575184.069740:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488865792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575184.069748:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575184.069751:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575184.069754:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488865792:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575184.069759:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488865920:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575184.069775:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575184.069784:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575184/real 1788575184] req@ffff94d408d3df80 x1875450488865920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575184.069794:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488865920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575184.069801:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575184.069804:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575184.069807:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488865920:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575184.581273:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575185.029361:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575185.605282:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575186.053315:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575186.629274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575187.077326:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575187.653267:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575188.101368:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575188.677305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575189.125321:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575189.189310:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575189.189317:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575189.189321:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575189.189328:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6337 00000100:00080000:2.0:1788575189.189336:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575189.189361:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575189.189391:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488866048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575189.189413:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575189.189415:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575189.189416:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575189.189423:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6337 00000100:00080000:2.0:1788575189.189425:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575189.189428:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575189.189435:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575189.189443:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488866176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575189.189451:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575189.189452:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488866048:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788575189.189454:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575189.189456:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575189.189462:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6337 00000100:00080000:2.0:1788575189.189463:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575189.189465:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575189.189468:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575189.189476:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488866304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575189.189481:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575189.189484:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575189.189485:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575189.189488:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6337 00000100:00080000:2.0:1788575189.189496:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575189.189498:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575189.189502:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788575189.189511:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575189.189511:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488866432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575189.189519:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575189.189521:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575189.189523:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575189.189525:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6337 00000100:00080000:2.0:1788575189.189530:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575189.189532:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575189.189535:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575189.189544:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488866560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575189.189546:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575189/real 1788575189] req@ffff94d408c61180 x1875450488866048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575189.189568:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488866048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575189.189576:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575189.189579:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575189.189582:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488866048:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575189.189600:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488866176:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575189.189613:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575189.189620:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575189/real 1788575189] req@ffff94d408c62d80 x1875450488866176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575189.189628:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488866176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575189.189633:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575189.189635:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575189.189637:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488866176:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575189.189641:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488866304:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575189.189650:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575189.189655:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575189/real 1788575189] req@ffff94d408c60000 x1875450488866304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575189.189662:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488866304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575189.189666:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575189.189668:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575189.189669:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488866304:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575189.189673:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488866432:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575189.189682:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575189.189686:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575189/real 1788575189] req@ffff94d408c63800 x1875450488866432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575189.189693:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488866432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575189.189697:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575189.189699:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575189.189700:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488866432:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575189.189703:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488866560:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575189.189711:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575189.189721:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575189/real 1788575189] req@ffff94d408c62680 x1875450488866560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575189.189727:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488866560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575189.189732:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575189.189734:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575189.189735:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488866560:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575189.701274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575190.149372:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575190.725287:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575191.173354:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575191.749308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575192.197341:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575192.773258:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575193.221344:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575193.797298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575194.245336:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575194.309300:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575194.309308:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575194.309311:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575194.309318:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6342 00000100:00080000:0.0:1788575194.309326:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575194.309332:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575194.309359:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488866688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575194.309380:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575194.309382:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575194.309384:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575194.309386:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6342 00000100:00080000:0.0:1788575194.309388:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575194.309390:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575194.309398:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575194.309405:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488866816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575194.309412:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575194.309416:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788575194.309417:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488866688:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788575194.309418:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575194.309424:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6342 00000100:00080000:0.0:1788575194.309427:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575194.309429:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575194.309431:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575194.309439:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488866944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575194.309444:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575194.309446:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575194.309448:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575194.309450:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6342 00000100:00080000:0.0:1788575194.309453:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575194.309455:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575194.309458:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575194.309465:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488867072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575194.309472:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788575194.309472:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788575194.309477:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575194.309479:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575194.309483:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6342 00000100:00080000:0.0:1788575194.309486:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575194.309488:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575194.309491:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575194.309498:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488867200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575194.309504:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575194/real 1788575194] req@ffff94d408d3ce00 x1875450488866688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575194.309518:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488866688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575194.309525:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575194.309528:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575194.309531:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488866688:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575194.309548:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488866816:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575194.309560:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575194.309566:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575194/real 1788575194] req@ffff94d408d3d500 x1875450488866816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575194.309573:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488866816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575194.309577:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575194.309579:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575194.309581:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488866816:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575194.309584:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488866944:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575194.309594:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575194.309599:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575194/real 1788575194] req@ffff94d408d3ca80 x1875450488866944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575194.309605:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488866944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575194.309619:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575194.309620:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575194.309625:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488866944:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575194.309631:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488867072:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575194.309652:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575194.309662:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575194/real 1788575194] req@ffff94d408d3ea00 x1875450488867072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575194.309674:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488867072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575194.309681:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575194.309684:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575194.309687:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488867072:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575194.309693:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488867200:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575194.309710:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575194.309721:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575194/real 1788575194] req@ffff94d408d3e680 x1875450488867200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575194.309738:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488867200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575194.309745:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575194.309746:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575194.309748:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488867200:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575194.821261:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575195.269350:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575195.845303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575196.293357:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575196.869264:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575197.317327:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575197.893280:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575198.341378:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575198.917270:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575199.365368:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575199.429302:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575199.429313:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575199.429318:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575199.429327:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6347 00000100:00080000:1.0:1788575199.429337:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575199.429343:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575199.429381:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488867328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575199.429399:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575199.429401:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575199.429403:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575199.429405:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6347 00000100:00080000:1.0:1788575199.429409:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575199.429412:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575199.429419:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575199.429427:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd880 x1875450488867456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575199.429432:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488867328:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788575199.429436:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575199.429441:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575199.429442:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575199.429445:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6347 00000100:00080000:1.0:1788575199.429447:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575199.429449:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575199.429452:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575199.429459:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488867584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575199.429465:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575199.429467:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575199.429468:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575199.429470:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6347 00000100:00080000:1.0:1788575199.429472:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575199.429475:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575199.429478:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575199.429483:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488867712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575199.429492:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575199.429494:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575199.429495:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575199.429497:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6347 00000100:00080000:1.0:1788575199.429500:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788575199.429504:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788575199.429506:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575199.429510:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575199.429516:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488867840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575199.429536:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575199/real 1788575199] req@ffff94d4049bca80 x1875450488867328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575199.429549:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488867328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575199.429556:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575199.429559:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575199.429561:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488867328:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575199.429578:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488867456:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575199.429593:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575199.429603:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575199/real 1788575199] req@ffff94d4049bd880 x1875450488867456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575199.429616:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd880 x1875450488867456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575199.429624:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575199.429628:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575199.429631:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488867456:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575199.429637:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488867584:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575199.429656:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575199.429668:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575199/real 1788575199] req@ffff94d4049be300 x1875450488867584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575199.429679:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488867584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575199.429686:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575199.429688:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575199.429690:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488867584:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575199.429693:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488867712:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575199.429707:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575199.429713:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575199/real 1788575199] req@ffff94d4049bf100 x1875450488867712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575199.429719:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488867712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575199.429742:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575199.429745:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575199.429747:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488867712:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575199.429750:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488867840:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575199.429762:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575199.429768:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575199/real 1788575199] req@ffff94d4049be680 x1875450488867840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575199.429775:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488867840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575199.429779:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575199.429781:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575199.429783:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488867840:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575199.941268:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575200.389349:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575200.965286:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575201.413315:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575201.989302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575202.437340:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575203.013289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575203.461316:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575204.037303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575204.485307:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575204.549327:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575204.549336:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575204.549340:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575204.549349:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6352 00000100:00080000:2.0:1788575204.549360:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575204.549367:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575204.549401:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62300 x1875450488867968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575204.549429:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575204.549433:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575204.549435:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575204.549438:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6352 00000100:00080000:2.0:1788575204.549442:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575204.549446:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575204.549455:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575204.549464:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488867968:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575204.549469:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60380 x1875450488868096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575204.549494:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575204.549498:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575204.549500:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575204.549504:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6352 00000100:00080000:2.0:1788575204.549509:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575204.549512:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575204.549517:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575204.549529:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60700 x1875450488868224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575204.549535:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788575204.549543:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575204.549546:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575204.549548:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575204.549553:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6352 00000100:00080000:2.0:1788575204.549558:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575204.549562:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575204.549567:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575204.549572:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575204/real 1788575204] req@ffff94d408c62300 x1875450488867968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575204.549581:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488868352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575204.549588:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62300 x1875450488867968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575204.549598:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575204.549601:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575204.549602:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575204.549606:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:2.0:1788575204.549607:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575204.549610:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488867968:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788575204.549615:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6352 00000100:00080000:2.0:1788575204.549620:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575204.549624:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575204.549630:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575204.549631:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488868096:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575204.549658:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488868480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575204.549660:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575204.549671:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575204/real 1788575204] req@ffff94d408c60380 x1875450488868096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575204.549680:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60380 x1875450488868096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575204.549686:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575204.549689:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575204.549691:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488868096:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575204.549696:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488868224:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575204.549707:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575204.549712:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575204/real 1788575204] req@ffff94d408c60700 x1875450488868224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575204.549718:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60700 x1875450488868224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575204.549723:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575204.549724:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575204.549726:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488868224:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575204.549729:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488868352:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575204.549739:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575204.549743:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575204/real 1788575204] req@ffff94d408c62680 x1875450488868352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575204.549754:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488868352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575204.549758:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575204.549759:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575204.549761:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488868352:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575204.549769:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488868480:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575204.549778:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575204.549783:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575204/real 1788575204] req@ffff94d408c63800 x1875450488868480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575204.549795:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488868480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575204.549799:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575204.549801:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575204.549803:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488868480:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575205.061286:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575205.509330:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575206.085284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575206.533345:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575207.109280:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575207.557385:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575208.133296:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575208.581366:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575209.157274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575209.605338:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575209.669405:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575209.669415:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575209.669419:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575209.669428:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6357 00000100:00080000:0.0:1788575209.669438:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575209.669446:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575209.669481:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f800 x1875450488868608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575209.669508:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575209.669512:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575209.669514:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575209.669518:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6357 00000100:00080000:0.0:1788575209.669522:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575209.669526:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575209.669534:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575209.669538:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488868608:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575209.669549:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f100 x1875450488868736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575209.669562:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575209.669565:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575209.669567:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575209.669577:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6357 00000100:00080000:0.0:1788575209.669582:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575209.669588:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:2.0:1788575209.669588:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:0.0:1788575209.669596:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575209.669611:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488868864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575209.669618:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575209/real 1788575209] req@ffff94d408d3f800 x1875450488868608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575209.669624:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575209.669633:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575209.669637:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575209.669643:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6357 00000100:00080000:0.0:1788575209.669647:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575209.669651:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575209.669653:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f800 x1875450488868608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788575209.669658:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788575209.669662:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575209.669665:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575209.669668:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488868608:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575209.669673:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488868992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575209.669684:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788575209.669688:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488868736:192.168.202.125@tcp:38:kworker.0 00000100:00080000:0.0:1788575209.669689:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575209.669694:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575209.669701:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6357 00000100:00080000:0.0:1788575209.669705:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788575209.669709:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788575209.669710:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575209.669718:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575209.669726:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575209/real 1788575209] req@ffff94d408d3f100 x1875450488868736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575209.669733:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488869120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575209.669738:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f100 x1875450488868736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575209.669754:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575209.669757:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575209.669760:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488868736:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575209.669764:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488868864:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575209.669776:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575209.669782:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575209/real 1788575209] req@ffff94d408d3e680 x1875450488868864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575209.669789:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488868864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575209.669797:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575209.669799:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575209.669801:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488868864:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575209.669803:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488868992:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575209.669813:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575209.669817:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575209/real 1788575209] req@ffff94d408d3ea00 x1875450488868992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575209.669823:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488868992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575209.669828:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575209.669829:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575209.669831:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488868992:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575209.669839:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488869120:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575209.669847:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575209.669852:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575209/real 1788575209] req@ffff94d408d3ca80 x1875450488869120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575209.669857:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488869120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575209.669862:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575209.669863:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575209.669865:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488869120:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575210.181309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575210.629353:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575211.205311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575211.653389:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575212.229272:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575212.677329:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575213.253320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575213.701359:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575214.277262:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575214.725303:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575214.789295:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575214.789303:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575214.789306:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575214.789314:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6363 00000100:00080000:0.0:1788575214.789322:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575214.789328:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575214.789357:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488869248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575214.789377:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575214.789380:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575214.789381:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575214.789383:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6363 00000100:00080000:0.0:1788575214.789385:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575214.789387:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575214.789394:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575214.789404:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488869376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575214.789414:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575214.789416:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575214.789418:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788575214.789421:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488869248:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788575214.789423:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6363 00000100:00080000:0.0:1788575214.789428:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575214.789432:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575214.789436:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575214.789446:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488869504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575214.789451:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575214.789453:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575214.789455:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575214.789458:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6363 00000100:00080000:0.0:1788575214.789460:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575214.789463:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575214.789467:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575214.789472:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488869632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575214.789478:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575214.789479:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575214.789480:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575214.789483:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6363 00000100:00080000:0.0:1788575214.789485:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575214.789487:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575214.789490:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575214.789498:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488869760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575214.789511:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488869376:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575214.789535:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488869504:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575214.789552:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488869632:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575214.789566:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488869760:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575215.301300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575215.749342:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575216.325313:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575216.773391:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575217.349295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575217.797427:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788575217.861474:0:2392:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:2.0:1788575217.861511:0:2392:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788575217.861530:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788575217.861562:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788575217.861568:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788575217.861574:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788575217.861580:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:0.0:1788575217.861691:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575214/real 1788575217] req@ffff94d408d3ce00 x1875450488869376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575217.861754:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488869376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575217.861773:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575217.861782:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575217.861791:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488869376:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575217.861807:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575214/real 1788575217] req@ffff94d408d3df80 x1875450488869504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575217.861826:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488869504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575217.861839:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575217.861844:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575217.861850:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488869504:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575217.861862:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575214/real 1788575217] req@ffff94d408d3d180 x1875450488869632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575217.861880:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488869632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575217.861890:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575217.861895:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575217.861900:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488869632:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575217.861909:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575214/real 1788575217] req@ffff94d408d3c000 x1875450488869760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575217.861943:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488869760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575217.861955:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575217.861960:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575217.861966:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488869760:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575217.861977:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575214/real 1788575217] req@ffff94d408d3d500 x1875450488869248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575217.861997:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488869248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575217.862007:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575217.862013:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575217.862018:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488869248:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575218.373317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575218.821403:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575219.397314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575219.845352:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575219.909283:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575219.909293:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575219.909297:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575219.909308:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6368 00000100:00080000:0.0:1788575219.909319:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575219.909327:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575219.909359:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488869888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575219.909390:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575219.909394:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575219.909396:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575219.909400:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6368 00000100:00080000:0.0:1788575219.909403:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575219.909408:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575219.909418:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575219.909432:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488870016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575219.909444:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488869888:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788575219.909446:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575219.909454:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575219.909458:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575219.909465:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6368 00000100:00080000:0.0:1788575219.909468:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575219.909472:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575219.909477:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575219.909489:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488870144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575219.909500:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575219.909503:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575219.909505:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575219.909509:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6368 00000100:00080000:0.0:1788575219.909514:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575219.909517:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:2.0:1788575219.909523:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:0.0:1788575219.909524:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575219.909540:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488870272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575219.909552:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575219.909555:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575219.909558:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575219.909564:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6368 00000100:00100000:2.0:1788575219.909565:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575219/real 1788575219] req@ffff94d408d3d500 x1875450488869888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575219.909569:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575219.909577:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575219.909587:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575219.909595:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488869888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575219.909602:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488870400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575219.909616:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575219.909624:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575219.909630:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488869888:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575219.909657:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488870016:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575219.909682:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575219.909696:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575219/real 1788575219] req@ffff94d408d3c000 x1875450488870016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575219.909709:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488870016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575219.909718:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575219.909721:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575219.909725:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488870016:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575219.909731:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488870144:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575219.909750:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575219.909761:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575219/real 1788575219] req@ffff94d408d3d180 x1875450488870144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575219.909781:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488870144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575219.909790:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575219.909793:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575219.909797:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488870144:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575219.909804:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488870272:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575219.909824:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575219.909835:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575219/real 1788575219] req@ffff94d408d3df80 x1875450488870272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575219.909866:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488870272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575219.909875:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575219.909878:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575219.909882:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488870272:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575219.909888:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488870400:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575219.909908:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575219.909919:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575219/real 1788575219] req@ffff94d408d3ce00 x1875450488870400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575219.909930:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488870400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575219.909944:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575219.909947:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575219.909951:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488870400:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575220.421336:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575220.869402:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575221.445345:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575221.893384:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575222.469278:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575222.917337:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575223.493390:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575223.941356:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575224.517384:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575224.965357:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575225.029301:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575225.029309:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575225.029313:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575225.029321:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6373 00000100:00080000:1.0:1788575225.029331:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575225.029337:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575225.029372:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488870528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575225.029400:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575225.029404:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575225.029406:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575225.029410:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6373 00000100:00080000:1.0:1788575225.029414:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575225.029417:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575225.029426:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575225.029438:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488870656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575225.029447:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788575225.029449:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488870528:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788575225.029450:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575225.029454:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575225.029462:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6373 00000100:00080000:1.0:1788575225.029464:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575225.029466:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575225.029469:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575225.029479:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488870784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575225.029485:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575225.029488:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575225.029489:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575225.029491:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6373 00000100:00080000:1.0:1788575225.029494:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575225.029497:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575225.029500:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575225.029506:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488870912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575225.029512:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575225.029514:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575225.029517:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788575225.029522:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788575225.029523:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6373 00000100:00080000:1.0:1788575225.029528:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575225.029530:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575225.029534:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575225.029541:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488871040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575225.029575:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575225/real 1788575225] req@ffff94d4049bdc00 x1875450488870528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575225.029596:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488870528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575225.029606:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575225.029610:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575225.029615:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488870528:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575225.029652:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488870656:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575225.029675:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575225.029688:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575225/real 1788575225] req@ffff94d4049bd500 x1875450488870656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575225.029708:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488870656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575225.029717:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575225.029720:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575225.029723:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488870656:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575225.029729:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488870784:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575225.029747:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575225.029759:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575225/real 1788575225] req@ffff94d4049bf800 x1875450488870784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575225.029769:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488870784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575225.029777:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575225.029780:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575225.029783:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488870784:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575225.029789:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488870912:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575225.029805:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575225.029814:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575225/real 1788575225] req@ffff94d4049bed80 x1875450488870912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575225.029825:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488870912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575225.029833:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575225.029836:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575225.029839:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488870912:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575225.029845:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488871040:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575225.029862:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575225.029873:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575225/real 1788575225] req@ffff94d4049bc700 x1875450488871040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575225.029884:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488871040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575225.029891:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575225.029894:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575225.029898:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488871040:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575225.541351:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575225.989426:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575226.565299:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575227.013369:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575227.589408:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575228.037435:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575228.613943:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575229.061448:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575229.637351:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575230.085340:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575230.149365:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575230.149378:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575230.149382:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575230.149392:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6378 00000100:00080000:2.0:1788575230.149405:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575230.149413:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575230.149482:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488871168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575230.149519:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575230.149523:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575230.149525:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575230.149528:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6378 00000100:00080000:2.0:1788575230.149531:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575230.149535:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575230.149543:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575230.149559:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488871296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575230.149567:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575230.149570:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575230.149571:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575230.149575:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6378 00000100:00080000:2.0:1788575230.149577:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575230.149580:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575230.149583:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575230.149590:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488871168:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575230.149593:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488871424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575230.149606:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575230.149609:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575230.149610:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575230.149612:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6378 00000100:00080000:2.0:1788575230.149614:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575230.149616:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575230.149621:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575230.149629:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488871552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575230.149636:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575230.149639:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575230.149640:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575230.149642:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6378 00000100:00080000:2.0:1788575230.149644:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575230.149646:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575230.149649:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575230.149656:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488871680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575230.149685:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575230.149736:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575230/real 1788575230] req@ffff94d408c60000 x1875450488871168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575230.149759:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488871168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575230.149771:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575230.149775:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575230.149779:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488871168:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575230.149788:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488871296:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575230.149816:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575230.149831:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575230/real 1788575230] req@ffff94d408c62d80 x1875450488871296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575230.149841:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488871296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575230.149849:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575230.149852:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575230.149856:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488871296:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575230.149888:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488871424:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575230.149912:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575230.149936:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575230/real 1788575230] req@ffff94d408c61180 x1875450488871424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575230.149945:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488871424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575230.149952:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575230.149955:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575230.149957:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488871424:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575230.149962:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488871552:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575230.149977:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575230.149984:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575230/real 1788575230] req@ffff94d408c63480 x1875450488871552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575230.149991:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488871552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575230.149997:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575230.149999:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575230.150002:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488871552:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575230.150005:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488871680:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575230.150015:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575230.150021:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575230/real 1788575230] req@ffff94d408c61f80 x1875450488871680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575230.150028:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488871680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575230.150033:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575230.150035:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575230.150037:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488871680:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575230.661290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575231.109503:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575231.685328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575232.133399:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575232.709371:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575233.157467:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575233.733321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575234.181453:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575234.757339:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575235.205444:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575235.269284:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575235.269297:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575235.269302:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575235.269313:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6383 00000100:00080000:2.0:1788575235.269327:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575235.269335:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575235.269385:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488871808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575235.269425:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575235.269429:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575235.269431:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575235.269436:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6383 00000100:00080000:2.0:1788575235.269440:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575235.269445:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575235.269455:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575235.269469:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488871936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575235.269479:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575235.269482:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575235.269483:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575235.269486:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6383 00000100:00080000:2.0:1788575235.269489:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575235.269492:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575235.269496:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575235.269505:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488872064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575235.269517:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575235.269520:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1788575235.269521:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488871808:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788575235.269523:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575235.269545:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6383 00000100:00080000:2.0:1788575235.269550:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575235.269557:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575235.269565:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575235.269584:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488872192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575235.269592:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575235.269595:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575235.269597:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575235.269600:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6383 00000100:00080000:2.0:1788575235.269604:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788575235.269609:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788575235.269613:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575235.269621:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575235.269634:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488872320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575235.269663:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575235/real 1788575235] req@ffff94d408c62a00 x1875450488871808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575235.269679:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488871808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575235.269692:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575235.269695:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575235.269698:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488871808:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575235.269705:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488871936:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575235.269721:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575235.269729:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575235/real 1788575235] req@ffff94d408c63b80 x1875450488871936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575235.269737:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488871936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575235.269743:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575235.269745:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575235.269747:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488871936:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575235.269768:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488872064:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575235.269778:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575235.269785:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575235/real 1788575235] req@ffff94d408c60a80 x1875450488872064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575235.269792:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488872064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575235.269798:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575235.269800:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575235.269802:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488872064:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575235.269806:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488872192:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575235.269828:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575235.269844:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575235/real 1788575235] req@ffff94d408c61500 x1875450488872192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575235.269853:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488872192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575235.269858:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575235.269860:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575235.269862:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488872192:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575235.269866:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488872320:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575235.269878:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575235.269891:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575235/real 1788575235] req@ffff94d408c61880 x1875450488872320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575235.269898:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488872320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575235.269903:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575235.269906:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575235.269908:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488872320:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575235.781313:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575236.229467:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575236.805367:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575237.253450:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575237.829341:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575238.277394:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575238.853336:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575239.301459:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575239.877317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575240.325422:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575240.389338:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575240.389349:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575240.389354:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575240.389365:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6388 00000100:00080000:2.0:1788575240.389378:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575240.389386:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575240.389434:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63100 x1875450488872448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575240.389467:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575240.389470:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575240.389472:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575240.389475:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6388 00000100:00080000:2.0:1788575240.389477:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575240.389480:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575240.389490:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575240.389506:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60e00 x1875450488872576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575240.389519:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575240.389525:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488872448:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788575240.389526:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575240.389534:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575240.389543:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6388 00000100:00080000:2.0:1788575240.389547:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575240.389551:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575240.389557:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575240.389569:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488872704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575240.389577:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575240.389579:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575240.389581:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575240.389586:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6388 00000100:00080000:2.0:1788575240.389591:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575240.389596:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:0.0:1788575240.389597:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:2.0:1788575240.389607:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575240.389622:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488872832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575240.389630:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575240.389633:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575240.389636:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575240.389639:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6388 00000100:00080000:2.0:1788575240.389642:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575240.389648:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575240.389650:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575240/real 1788575240] req@ffff94d408c63100 x1875450488872448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788575240.389653:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575240.389665:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488872960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575240.389706:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63100 x1875450488872448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575240.389723:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575240.389730:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575240.389735:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488872448:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575240.389763:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488872576:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575240.389792:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575240.389806:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575240/real 1788575240] req@ffff94d408c60e00 x1875450488872576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575240.389815:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60e00 x1875450488872576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575240.389822:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575240.389824:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575240.389826:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488872576:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575240.389831:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488872704:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575240.389846:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575240.389853:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575240/real 1788575240] req@ffff94d408c61c00 x1875450488872704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575240.389865:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488872704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575240.389871:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575240.389873:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575240.389875:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488872704:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575240.389878:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488872832:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575240.389891:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575240.389897:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575240/real 1788575240] req@ffff94d408c61880 x1875450488872832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575240.389904:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488872832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575240.389909:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575240.389911:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575240.389913:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488872832:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575240.389917:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488872960:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575240.389926:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575240.389932:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575240/real 1788575240] req@ffff94d408c61500 x1875450488872960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575240.389939:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488872960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575240.389948:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575240.389950:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575240.389952:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488872960:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575240.901318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575241.349455:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575241.925358:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575242.373372:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575242.949365:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575243.397447:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575243.973290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575244.421372:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575244.997314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575245.445381:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575245.509340:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575245.509352:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575245.509357:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575245.509367:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6393 00000100:00080000:3.0:1788575245.509379:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575245.509387:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575245.509439:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488873088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575245.509469:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575245.509474:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575245.509476:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575245.509480:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6393 00000100:00080000:3.0:1788575245.509484:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575245.509489:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575245.509503:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575245.509517:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488873216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575245.509529:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575245.509534:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575245.509537:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575245.509542:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6393 00000100:00100000:0.0:1788575245.509547:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488873088:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575245.509548:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575245.509558:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575245.509565:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575245.509581:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488873344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575245.509604:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575245.509608:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575245.509613:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575245.509616:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575245.509628:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6393 00000100:00080000:3.0:1788575245.509633:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788575245.509638:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575245/real 1788575245] req@ffff94d408c57480 x1875450488873088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575245.509642:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575245.509650:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575245.509666:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488873088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575245.509672:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488873472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575245.509678:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575245.509687:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575245.509693:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488873088:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575245.509695:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575245.509701:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1788575245.509702:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488873216:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788575245.509706:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575245.509714:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6393 00000100:00080000:3.0:1788575245.509719:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575245.509724:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:0.0:1788575245.509730:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:3.0:1788575245.509731:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575245.509748:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575245/real 1788575245] req@ffff94d408c54000 x1875450488873216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575245.509750:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488873600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575245.509806:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488873216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575245.509821:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575245.509824:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575245.509827:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488873216:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575245.509849:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488873344:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575245.509872:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575245.509881:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575245/real 1788575245] req@ffff94d408c55f80 x1875450488873344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575245.509889:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488873344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575245.509895:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575245.509897:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575245.509899:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488873344:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575245.509903:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488873472:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575245.509915:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575245.509921:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575245/real 1788575245] req@ffff94d408c56300 x1875450488873472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575245.509928:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488873472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575245.509932:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575245.509934:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575245.509937:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488873472:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575245.509940:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488873600:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575245.509949:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575245.509955:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575245/real 1788575245] req@ffff94d408c54700 x1875450488873600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575245.509962:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488873600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575245.509966:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575245.509968:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575245.509970:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488873600:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575246.021258:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575246.469345:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575247.045321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575247.493345:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575248.069280:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575248.517388:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575249.093325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575249.541401:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575250.117311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575250.565374:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575250.629356:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575250.629371:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575250.629377:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575250.629391:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6398 00000100:00080000:3.0:1788575250.629408:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575250.629418:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575250.629473:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488873728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575250.629515:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575250.629520:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575250.629523:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575250.629528:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6398 00000100:00080000:3.0:1788575250.629532:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575250.629537:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575250.629548:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575250.629565:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488873856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575250.629593:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575250.629599:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788575250.629601:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488873728:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575250.629603:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575250.629613:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6398 00000100:00080000:3.0:1788575250.629620:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575250.629625:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575250.629632:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575250.629647:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488873984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575250.629660:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575250.629664:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575250.629666:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575250.629670:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6398 00000100:00080000:3.0:1788575250.629675:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575250.629680:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575250.629686:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788575250.629697:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575250.629702:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488874112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575250.629714:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575250.629718:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575250.629721:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575250.629725:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6398 00000100:00080000:3.0:1788575250.629731:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575250.629736:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575250.629742:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575250.629754:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488874240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575250.629771:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575250/real 1788575250] req@ffff94d408c55180 x1875450488873728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575250.629796:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488873728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575250.629810:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575250.629815:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575250.629819:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488873728:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575250.629851:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488873856:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575250.629901:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575250.629920:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575250/real 1788575250] req@ffff94d408c55880 x1875450488873856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575250.629935:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488873856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575250.629944:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575250.629948:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575250.629951:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488873856:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575250.629958:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488873984:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575250.629985:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575250.629999:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575250/real 1788575250] req@ffff94d408c54380 x1875450488873984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575250.630014:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488873984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575250.630025:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575250.630030:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575250.630034:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488873984:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575250.630044:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488874112:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575250.630073:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575250.630087:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575250/real 1788575250] req@ffff94d408c57800 x1875450488874112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575250.630107:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488874112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575250.630116:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575250.630119:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575250.630123:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488874112:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575250.630135:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488874240:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575250.630159:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575250.630171:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575250/real 1788575250] req@ffff94d408c56a00 x1875450488874240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575250.630184:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488874240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575250.630193:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575250.630197:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575250.630237:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488874240:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575251.141307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575251.589389:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575252.165380:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575252.613366:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575253.189301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575253.637333:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575254.213334:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575254.661473:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575255.237333:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575255.685408:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575255.749340:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575255.749351:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575255.749355:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575255.749378:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6404 00000100:00080000:0.0:1788575255.749389:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575255.749397:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575255.749436:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488874368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575255.749467:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575255.749471:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575255.749473:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575255.749477:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6404 00000100:00080000:0.0:1788575255.749482:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575255.749488:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575255.749497:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575255.749508:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488874496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575255.749518:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575255.749522:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788575255.749522:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488874368:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788575255.749530:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575255.749543:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6404 00000100:00080000:0.0:1788575255.749546:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575255.749548:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575255.749551:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575255.749562:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488874624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575255.749569:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575255.749572:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575255.749573:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575255.749576:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6404 00000100:00080000:0.0:1788575255.749579:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575255.749582:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575255.749585:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575255.749594:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488874752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575255.749599:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575255.749601:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575255.749603:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575255.749605:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6404 00000100:00080000:0.0:1788575255.749608:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575255.749611:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575255.749614:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788575255.749620:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575255.749624:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488874880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575255.749666:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575255/real 1788575255] req@ffff94d408d3f480 x1875450488874368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575255.749682:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488874368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575255.749692:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575255.749695:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575255.749699:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488874368:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575255.749725:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488874496:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575255.749743:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575255.749752:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575255/real 1788575255] req@ffff94d408d3c700 x1875450488874496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575255.749760:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488874496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575255.749767:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575255.749769:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575255.749772:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488874496:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575255.749776:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488874624:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575255.749787:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575255.749793:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575255/real 1788575255] req@ffff94d408d3e300 x1875450488874624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575255.749801:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488874624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575255.749807:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575255.749809:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575255.749810:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488874624:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575255.749814:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488874752:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575255.749836:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575255.749843:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575255/real 1788575255] req@ffff94d408d3dc00 x1875450488874752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575255.749851:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488874752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575255.749856:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575255.749859:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575255.749861:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488874752:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575255.749865:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488874880:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575255.749877:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575255.749883:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575255/real 1788575255] req@ffff94d408d3c380 x1875450488874880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575255.749896:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488874880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575255.749901:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575255.749903:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575255.749904:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488874880:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575256.261372:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575256.709463:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575257.285337:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575257.733473:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575258.309320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575258.757368:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575259.333335:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575259.781352:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575260.357307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575260.805393:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575260.869339:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575260.869351:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575260.869356:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575260.869367:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6409 00000100:00080000:0.0:1788575260.869380:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575260.869388:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575260.869435:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488875008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575260.869466:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575260.869469:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575260.869471:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575260.869474:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6409 00000100:00080000:0.0:1788575260.869476:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575260.869479:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575260.869489:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575260.869501:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ed80 x1875450488875136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575260.869508:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575260.869511:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575260.869513:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575260.869517:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6409 00000100:00080000:0.0:1788575260.869520:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575260.869523:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575260.869526:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575260.869534:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488875264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575260.869541:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575260.869544:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575260.869545:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575260.869550:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6409 00000100:00080000:0.0:1788575260.869556:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575260.869561:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575260.869562:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488875008:192.168.202.125@tcp:250:kworker.0 00000008:00100000:0.0:1788575260.869566:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575260.869578:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488875392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575260.869585:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575260.869588:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575260.869589:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575260.869592:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6409 00000100:00080000:0.0:1788575260.869594:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575260.869598:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575260.869602:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575260.869610:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488875520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788575260.869651:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575260.869690:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575260/real 1788575260] req@ffff94d408d3d880 x1875450488875008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575260.869711:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488875008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575260.869733:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575260.869739:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575260.869744:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488875008:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575260.869754:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488875136:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575260.869778:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575260.869792:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575260/real 1788575260] req@ffff94d408d3ed80 x1875450488875136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575260.869807:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ed80 x1875450488875136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575260.869817:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575260.869821:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575260.869833:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488875136:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575260.869862:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488875264:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575260.869885:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575260.869896:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575260/real 1788575260] req@ffff94d408d3c380 x1875450488875264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575260.869919:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488875264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575260.869929:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575260.869933:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575260.869937:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488875264:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575260.869944:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488875392:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575260.869966:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575260.869977:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575260/real 1788575260] req@ffff94d408d3dc00 x1875450488875392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575260.869990:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488875392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575260.870007:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575260.870011:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575260.870015:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488875392:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575260.870022:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488875520:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575260.870042:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575260.870054:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575260/real 1788575260] req@ffff94d408d3e300 x1875450488875520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575260.870067:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488875520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575260.870076:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575260.870080:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575260.870084:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488875520:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575261.381289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575261.829348:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575262.405304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575262.853414:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575263.429298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575263.877350:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575264.453331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575264.901396:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575265.477280:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575265.925341:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575265.989305:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575265.989328:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575265.989333:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575265.989346:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6414 00000100:00080000:2.0:1788575265.989360:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575265.989370:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575265.989428:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488875648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575265.989464:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575265.989470:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575265.989479:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575265.989485:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6414 00000100:00080000:2.0:1788575265.989489:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575265.989495:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575265.989507:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575265.989524:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488875776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575265.989528:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488875648:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788575265.989547:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575265.989552:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575265.989554:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575265.989560:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6414 00000100:00080000:2.0:1788575265.989564:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575265.989568:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575265.989574:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575265.989589:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488875904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575265.989611:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788575265.989612:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575265.989618:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575265.989621:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575265.989628:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6414 00000100:00080000:2.0:1788575265.989635:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575265.989640:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575265.989647:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575265.989653:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575265/real 1788575265] req@ffff94d408c60a80 x1875450488875648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575265.989671:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488875648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575265.989672:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488876032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575265.989691:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575265.989695:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575265.989698:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575265.989708:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575265.989709:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6414 00000100:00080000:0.0:1788575265.989717:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:2.0:1788575265.989718:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788575265.989724:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488875648:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788575265.989728:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575265.989736:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575265.989752:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488876160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575265.989754:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488875776:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575265.989782:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575265.989792:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575265/real 1788575265] req@ffff94d408c63b80 x1875450488875776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575265.989806:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488875776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575265.989815:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575265.989818:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575265.989822:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488875776:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575265.989829:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488875904:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575265.989849:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575265.989861:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575265/real 1788575265] req@ffff94d408c62a00 x1875450488875904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575265.989871:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488875904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575265.989877:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575265.989880:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575265.989882:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488875904:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575265.989887:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488876032:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575265.989903:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575265.989909:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575265/real 1788575265] req@ffff94d408c61f80 x1875450488876032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575265.989917:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488876032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575265.989922:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575265.989925:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575265.989927:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488876032:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575265.989962:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488876160:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575265.989978:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575265.989986:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575265/real 1788575265] req@ffff94d408c63480 x1875450488876160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575265.989993:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488876160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575265.989998:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575265.990001:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575265.990003:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488876160:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575266.501325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575266.949388:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575267.525371:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575267.973454:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575268.549350:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575268.997436:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575269.573368:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575270.021425:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575270.597307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575271.045386:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575271.109364:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575271.109375:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575271.109379:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575271.109390:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6419 00000100:00080000:0.0:1788575271.109403:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575271.109411:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575271.109458:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488876288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575271.109486:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575271.109490:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575271.109491:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575271.109494:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6419 00000100:00080000:0.0:1788575271.109496:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575271.109499:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575271.109509:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575271.109518:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488876416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575271.109525:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575271.109528:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575271.109530:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575271.109534:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6419 00000100:00080000:0.0:1788575271.109538:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788575271.109538:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488876288:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788575271.109542:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575271.109547:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575271.109559:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488876544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575271.109570:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575271.109573:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575271.109576:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575271.109581:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6419 00000100:00080000:0.0:1788575271.109585:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575271.109590:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575271.109597:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575271.109613:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488876672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575271.109624:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788575271.109629:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788575271.109631:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575271.109636:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575271.109641:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6419 00000100:00080000:0.0:1788575271.109647:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575271.109652:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575271.109659:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575271.109671:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488876800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575271.109693:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575271/real 1788575271] req@ffff94d408d3c700 x1875450488876288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575271.109726:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488876288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575271.109739:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575271.109744:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575271.109749:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488876288:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575271.109800:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488876416:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575271.109829:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575271.109845:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575271/real 1788575271] req@ffff94d408d3f480 x1875450488876416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575271.109859:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488876416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575271.109869:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575271.109872:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575271.109876:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488876416:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575271.109883:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488876544:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575271.109905:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575271.109918:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575271/real 1788575271] req@ffff94d408d3ce00 x1875450488876544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575271.109930:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488876544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575271.109946:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575271.109949:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575271.109953:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488876544:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575271.109960:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488876672:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575271.109983:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575271.110009:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575271/real 1788575271] req@ffff94d408d3df80 x1875450488876672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575271.110023:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488876672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575271.110032:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575271.110035:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575271.110039:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488876672:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575271.110046:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488876800:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575271.110066:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575271.110078:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575271/real 1788575271] req@ffff94d408d3d180 x1875450488876800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575271.110090:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488876800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575271.110098:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575271.110101:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575271.110105:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488876800:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575271.621318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575272.069448:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575272.645297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575273.093459:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575273.669332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575274.117422:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575274.693315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575275.141424:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575275.717349:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575276.165414:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575276.229304:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575276.229318:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575276.229322:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575276.229334:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6424 00000100:00080000:1.0:1788575276.229349:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575276.229358:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575276.229410:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488876928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575276.229440:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575276.229443:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575276.229444:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575276.229447:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6424 00000100:00080000:1.0:1788575276.229450:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575276.229453:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575276.229462:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575276.229473:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488877056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575276.229484:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488876928:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788575276.229486:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575276.229492:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575276.229494:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575276.229498:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6424 00000100:00080000:1.0:1788575276.229500:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575276.229502:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575276.229504:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575276.229512:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488877184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575276.229517:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575276.229519:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575276.229520:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575276.229523:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6424 00000100:00080000:1.0:1788575276.229525:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575276.229527:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575276.229530:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575276.229536:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488877312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575276.229541:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575276.229543:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575276.229544:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575276.229553:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6424 00000100:00080000:1.0:1788575276.229557:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575276.229559:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575276.229563:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788575276.229571:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575276.229575:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488877440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575276.229625:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575276/real 1788575276] req@ffff94d4049bc000 x1875450488876928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575276.229644:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488876928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575276.229654:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575276.229658:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575276.229662:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488876928:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575276.229684:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488877056:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575276.229703:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575276.229713:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575276/real 1788575276] req@ffff94d4049bfb80 x1875450488877056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575276.229723:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488877056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575276.229730:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575276.229733:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575276.229737:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488877056:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575276.229742:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488877184:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575276.229755:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575276.229763:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575276/real 1788575276] req@ffff94d4049bd180 x1875450488877184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575276.229773:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488877184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575276.229779:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575276.229782:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575276.229785:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488877184:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575276.229790:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488877312:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575276.229809:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575276.229817:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575276/real 1788575276] req@ffff94d4049bce00 x1875450488877312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575276.229826:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488877312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575276.229833:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575276.229836:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575276.229838:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488877312:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575276.229843:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488877440:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575276.229857:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575276.229864:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575276/real 1788575276] req@ffff94d4049bea00 x1875450488877440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575276.229874:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488877440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575276.229880:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575276.229883:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575276.229886:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488877440:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575276.741360:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575277.189407:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575277.765357:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575278.213387:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575278.789372:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575279.237378:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575279.813367:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575280.261422:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575280.837336:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575281.285363:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575281.349308:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575281.349315:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575281.349319:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575281.349326:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6429 00000100:00080000:0.0:1788575281.349334:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575281.349341:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575281.349374:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488877568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575281.349401:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575281.349404:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575281.349405:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575281.349408:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6429 00000100:00080000:0.0:1788575281.349411:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575281.349414:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575281.349421:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575281.349431:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488877696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575281.349438:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575281.349441:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575281.349442:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575281.349457:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6429 00000100:00080000:0.0:1788575281.349461:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575281.349467:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575281.349473:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488877568:192.168.202.125@tcp:250:kworker.0 00000008:00100000:0.0:1788575281.349476:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575281.349498:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488877824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575281.349508:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575281.349512:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575281.349514:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575281.349518:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6429 00000100:00080000:0.0:1788575281.349522:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575281.349526:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575281.349531:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575281.349542:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488877952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575281.349550:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575281.349554:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575281.349555:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575281.349558:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6429 00000100:00080000:0.0:1788575281.349563:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575281.349567:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575281.349574:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575281.349586:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488877696:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575281.349598:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488878080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575281.349631:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488877824:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575281.349660:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488877952:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575281.349678:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488878080:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575281.861324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575282.309437:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575282.885302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575283.333355:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575283.909336:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575284.357392:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788575284.421582:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:2.0:1788575284.421642:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788575284.421664:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788575284.421710:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788575284.421721:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788575284.421733:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788575284.421744:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:0.0:1788575284.421755:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575281/real 1788575284] req@ffff94d408d3ca80 x1875450488877824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575284.421786:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488877824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575284.421800:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575284.421806:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575284.421810:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488877824:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575284.421819:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575281/real 1788575284] req@ffff94d408d3ea00 x1875450488877952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575284.421832:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488877952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575284.421841:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575284.421844:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575284.421848:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488877952:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575284.421855:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575281/real 1788575284] req@ffff94d408d3e680 x1875450488878080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575284.421867:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488878080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575284.421875:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575284.421879:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575284.421882:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488878080:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575284.421894:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575281/real 1788575284] req@ffff94d408d3c000 x1875450488877568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575284.421907:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488877568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575284.421914:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575284.421918:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575284.421922:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488877568:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575284.421930:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575281/real 1788575284] req@ffff94d408d3d500 x1875450488877696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575284.421942:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488877696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575284.421950:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575284.421954:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575284.421957:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488877696:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575284.933269:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575285.381382:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575285.957355:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575286.405443:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575286.469361:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575286.469374:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575286.469379:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575286.469392:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6434 00000100:00080000:0.0:1788575286.469405:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575286.469415:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575286.469460:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488878208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575286.469495:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575286.469500:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575286.469502:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575286.469508:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6434 00000100:00080000:0.0:1788575286.469513:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575286.469518:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575286.469529:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575286.469544:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488878336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575286.469554:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575286.469558:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575286.469560:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575286.469564:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6434 00000100:00080000:0.0:1788575286.469567:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575286.469571:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575286.469577:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575286.469588:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488878464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575286.469599:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575286.469603:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575286.469604:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575286.469607:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6434 00000100:00080000:0.0:1788575286.469611:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575286.469615:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575286.469621:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575286.469630:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488878592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575286.469639:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575286.469642:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575286.469645:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575286.469649:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6434 00000100:00080000:0.0:1788575286.469653:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575286.469658:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575286.469663:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575286.469674:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488878720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575286.469718:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488878208:192.168.202.125@tcp:250:kworker.0 00000800:00000100:0.0:1788575286.469819:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575286.469865:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575286/real 1788575286] req@ffff94d408d3d500 x1875450488878208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575286.469888:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488878208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575286.469900:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575286.469905:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575286.469910:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488878208:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575286.469920:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488878336:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575286.469948:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575286.469961:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575286/real 1788575286] req@ffff94d408d3c000 x1875450488878336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575286.469972:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488878336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575286.469981:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575286.469984:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575286.469986:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488878336:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575286.469990:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488878464:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575286.470008:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575286.470019:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575286/real 1788575286] req@ffff94d408d3e680 x1875450488878464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575286.470030:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488878464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575286.470038:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575286.470041:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575286.470044:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488878464:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575286.470049:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488878592:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575286.470068:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575286.470077:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575286/real 1788575286] req@ffff94d408d3ea00 x1875450488878592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575286.470089:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488878592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575286.470096:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575286.470099:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575286.470101:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488878592:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575286.470107:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488878720:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575286.470127:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575286.470138:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575286/real 1788575286] req@ffff94d408d3ca80 x1875450488878720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575286.470148:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488878720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575286.470154:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575286.470157:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575286.470160:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488878720:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575286.981305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575287.429372:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575288.005328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575288.453339:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575289.029317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575289.477400:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575290.054821:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575290.501326:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575291.079242:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575291.525382:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575291.589312:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575291.589323:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575291.589327:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575291.589338:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6439 00000100:00080000:1.0:1788575291.589349:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575291.589357:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575291.589399:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488878848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575291.589431:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575291.589434:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575291.589436:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575291.589438:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6439 00000100:00080000:1.0:1788575291.589441:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575291.589444:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575291.589452:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575291.589461:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488878976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575291.589470:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575291.589477:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788575291.589479:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488878848:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788575291.589479:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575291.589486:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6439 00000100:00080000:1.0:1788575291.589494:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575291.589501:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575291.589506:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575291.589522:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488879104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575291.589529:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575291.589532:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575291.589533:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575291.589536:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6439 00000100:00080000:1.0:1788575291.589551:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575291.589555:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575291.589558:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575291.589568:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488879232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575291.589574:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788575291.589586:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575291.589590:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575291.589591:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575291.589595:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6439 00000100:00080000:1.0:1788575291.589598:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575291.589601:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575291.589605:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575291.589614:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488879360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575291.589625:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575291/real 1788575291] req@ffff94d4049bf480 x1875450488878848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575291.589651:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488878848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575291.589659:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575291.589662:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575291.589666:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488878848:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575291.589689:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488878976:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575291.589717:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575291.589731:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575291/real 1788575291] req@ffff94d4049bc380 x1875450488878976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575291.589742:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc380 x1875450488878976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575291.589748:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575291.589751:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575291.589753:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488878976:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575291.589758:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488879104:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575291.589774:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575291.589781:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575291/real 1788575291] req@ffff94d4049bea00 x1875450488879104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575291.589789:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488879104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575291.589799:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575291.589801:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575291.589803:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488879104:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575291.589807:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488879232:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575291.589826:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575291.589838:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575291/real 1788575291] req@ffff94d4049bce00 x1875450488879232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575291.589851:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488879232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575291.589860:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575291.589863:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575291.589867:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488879232:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575291.589873:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488879360:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575291.589894:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575291.589906:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575291/real 1788575291] req@ffff94d4049bd180 x1875450488879360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575291.589919:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488879360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575291.589934:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575291.589938:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575291.589941:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488879360:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575292.101331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575292.549339:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575293.126159:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575293.573377:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575294.149674:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575294.597331:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575295.173295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575295.621325:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575296.197305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575296.645367:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575296.709342:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575296.709352:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575296.709356:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575296.709367:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6445 00000100:00080000:1.0:1788575296.709380:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575296.709389:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575296.709434:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488879488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575296.709462:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575296.709465:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575296.709467:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575296.709470:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6445 00000100:00080000:1.0:1788575296.709472:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575296.709475:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575296.709483:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575296.709492:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488879616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575296.709499:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575296.709502:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575296.709503:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575296.709505:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6445 00000100:00080000:1.0:1788575296.709508:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575296.709511:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575296.709514:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575296.709522:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488879488:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575296.709526:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488879744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575296.709535:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575296.709537:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575296.709538:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575296.709540:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6445 00000100:00080000:1.0:1788575296.709542:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575296.709544:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575296.709547:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575296.709556:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488879872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575296.709561:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575296.709563:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575296.709564:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575296.709565:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6445 00000100:00080000:1.0:1788575296.709567:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575296.709570:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575296.709574:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575296.709580:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488880000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575296.709603:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575296.709655:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575296/real 1788575296] req@ffff94d4049bfb80 x1875450488879488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575296.709671:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488879488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575296.709680:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575296.709683:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575296.709686:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488879488:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575296.709691:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488879616:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575296.709709:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575296.709718:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575296/real 1788575296] req@ffff94d4049bc000 x1875450488879616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575296.709725:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488879616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575296.709731:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575296.709734:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575296.709736:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488879616:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575296.709758:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488879744:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575296.709769:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575296.709775:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575296/real 1788575296] req@ffff94d4049bc700 x1875450488879744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575296.709782:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488879744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575296.709787:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575296.709790:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575296.709791:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488879744:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575296.709794:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488879872:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575296.709804:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575296.709809:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575296/real 1788575296] req@ffff94d4049bed80 x1875450488879872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575296.709815:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488879872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575296.709820:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575296.709822:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575296.709823:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488879872:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575296.709826:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488880000:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575296.709836:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575296.709841:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575296/real 1788575296] req@ffff94d4049bf800 x1875450488880000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575296.709848:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488880000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575296.709852:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575296.709854:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575296.709855:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488880000:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575297.221298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575297.669333:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575298.245287:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575298.693341:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575299.269306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575299.717424:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575300.293370:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575300.741407:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575301.317349:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575301.765368:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575301.829385:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575301.829397:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575301.829401:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575301.829411:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6450 00000100:00080000:2.0:1788575301.829427:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575301.829435:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575301.829480:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488880128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575301.829512:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575301.829517:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575301.829519:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575301.829522:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6450 00000100:00080000:2.0:1788575301.829526:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575301.829530:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575301.829540:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575301.829551:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488880256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575301.829556:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488880128:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788575301.829562:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575301.829565:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575301.829566:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575301.829569:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6450 00000100:00080000:2.0:1788575301.829572:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575301.829574:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575301.829577:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575301.829585:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488880384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575301.829590:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575301.829593:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575301.829594:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575301.829596:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6450 00000100:00080000:2.0:1788575301.829598:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575301.829600:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575301.829603:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575301.829610:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488880512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575301.829615:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575301.829618:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575301.829620:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575301.829622:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6450 00000100:00080000:2.0:1788575301.829625:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575301.829627:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575301.829630:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575301.829635:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488880640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575301.829642:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575301.829683:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575301/real 1788575301] req@ffff94d408c61180 x1875450488880128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575301.829697:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488880128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575301.829706:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575301.829709:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575301.829713:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488880128:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575301.829736:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488880256:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575301.829752:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575301.829759:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575301/real 1788575301] req@ffff94d408c62d80 x1875450488880256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575301.829767:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488880256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575301.829772:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575301.829774:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575301.829776:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488880256:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575301.829779:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488880384:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575301.829788:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575301.829792:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575301/real 1788575301] req@ffff94d408c60000 x1875450488880384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575301.829816:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488880384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575301.829822:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575301.829824:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575301.829826:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488880384:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575301.829829:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488880512:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575301.829843:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575301.829849:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575301/real 1788575301] req@ffff94d408c63800 x1875450488880512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575301.829856:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488880512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575301.829862:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575301.829866:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575301.829869:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488880512:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575301.829874:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488880640:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575301.829893:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575301.829904:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575301/real 1788575301] req@ffff94d408c62680 x1875450488880640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575301.829916:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488880640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575301.829921:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575301.829923:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575301.829925:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488880640:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575302.341300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575302.789363:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575303.365300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575303.813357:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575304.389267:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575304.837331:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575305.413292:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575305.861390:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575306.437290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575306.885364:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575306.949344:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575306.949352:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575306.949355:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575306.949364:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6455 00000100:00080000:1.0:1788575306.949373:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575306.949381:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575306.949421:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488880768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575306.949450:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575306.949454:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575306.949455:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575306.949458:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6455 00000100:00080000:1.0:1788575306.949462:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575306.949465:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575306.949472:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575306.949483:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488880896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575306.949491:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575306.949494:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575306.949495:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575306.949497:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6455 00000100:00080000:1.0:1788575306.949499:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575306.949501:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575306.949507:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575306.949509:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488880768:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575306.949516:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488881024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575306.949525:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575306.949527:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575306.949528:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575306.949532:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6455 00000100:00080000:1.0:1788575306.949535:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575306.949539:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575306.949545:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575306.949557:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488881152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575306.949568:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575306.949572:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575306.949574:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575306.949578:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6455 00000100:00080000:1.0:1788575306.949581:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575306.949585:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:0.0:1788575306.949591:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:1.0:1788575306.949591:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575306.949609:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488881280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575306.949664:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575306/real 1788575306] req@ffff94d4049bd500 x1875450488880768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575306.949686:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488880768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575306.949698:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575306.949702:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575306.949707:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488880768:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575306.949714:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488880896:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575306.949734:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575306.949744:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575306/real 1788575306] req@ffff94d4049bdc00 x1875450488880896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575306.949752:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488880896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575306.949758:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575306.949760:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575306.949762:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488880896:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575306.949783:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488881024:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575306.949814:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575306.949826:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575306/real 1788575306] req@ffff94d4049be680 x1875450488881024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575306.949843:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488881024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575306.949856:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575306.949859:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575306.949863:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488881024:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575306.949868:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488881152:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575306.949883:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575306.949891:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575306/real 1788575306] req@ffff94d4049bf100 x1875450488881152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575306.949899:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488881152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575306.949905:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575306.949907:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575306.949908:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488881152:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575306.949912:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488881280:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575306.949921:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575306.949927:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575306/real 1788575306] req@ffff94d4049be300 x1875450488881280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575306.949934:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488881280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575306.949943:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575306.949945:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575306.949947:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488881280:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575307.461309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575307.909359:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575308.485324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575308.933390:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575309.509294:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575309.957338:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575310.533317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575310.981371:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575311.557268:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575312.005321:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575312.069323:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575312.069331:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575312.069335:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575312.069343:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6460 00000100:00080000:2.0:1788575312.069352:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575312.069358:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575312.069391:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60700 x1875450488881408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575312.069412:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575312.069415:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575312.069416:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575312.069419:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6460 00000100:00080000:2.0:1788575312.069421:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575312.069423:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575312.069431:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575312.069439:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60380 x1875450488881536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575312.069452:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575312.069456:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488881408:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788575312.069457:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575312.069460:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575312.069466:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6460 00000100:00080000:2.0:1788575312.069468:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575312.069485:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575312.069490:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575312.069501:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62300 x1875450488881664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575312.069508:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575312.069511:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575312.069512:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575312.069515:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6460 00000100:00080000:2.0:1788575312.069524:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575312.069527:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:0.0:1788575312.069533:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:2.0:1788575312.069535:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575312.069546:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488881792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575312.069554:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575312.069556:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575312.069558:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575312.069560:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6460 00000100:00080000:2.0:1788575312.069569:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788575312.069573:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575312/real 1788575312] req@ffff94d408c60700 x1875450488881408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575312.069575:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575312.069581:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575312.069593:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488881920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575312.069596:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60700 x1875450488881408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575312.069618:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575312.069621:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575312.069624:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488881408:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575312.069644:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488881536:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575312.069660:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575312.069669:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575312/real 1788575312] req@ffff94d408c60380 x1875450488881536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575312.069677:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60380 x1875450488881536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575312.069683:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575312.069685:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575312.069687:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488881536:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575312.069691:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488881664:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575312.069701:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575312.069707:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575312/real 1788575312] req@ffff94d408c62300 x1875450488881664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575312.069714:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62300 x1875450488881664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575312.069719:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575312.069721:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575312.069723:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488881664:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575312.069726:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488881792:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575312.069736:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575312.069741:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575312/real 1788575312] req@ffff94d408c62680 x1875450488881792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575312.069749:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488881792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575312.069754:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575312.069756:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575312.069758:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488881792:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575312.069762:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488881920:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575312.069771:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575312.069776:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575312/real 1788575312] req@ffff94d408c63800 x1875450488881920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575312.069786:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488881920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575312.069794:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575312.069798:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575312.069801:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488881920:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575312.581265:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575313.029384:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575313.605292:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575314.053369:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575314.629257:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575315.077343:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575315.653321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575316.101388:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575316.677311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575317.125463:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575317.189274:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575317.189289:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575317.189294:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575317.189322:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6465 00000100:00080000:2.0:1788575317.189338:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575317.189347:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575317.189402:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488882048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575317.189430:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575317.189433:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575317.189435:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575317.189439:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6465 00000100:00080000:2.0:1788575317.189444:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575317.189449:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575317.189462:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575317.189472:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488882048:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575317.189480:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488882176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575317.189494:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575317.189510:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575317.189513:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575317.189517:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6465 00000100:00080000:2.0:1788575317.189522:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575317.189526:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575317.189533:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575317.189549:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488882304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575317.189557:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788575317.189566:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575317.189570:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575317.189572:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575317.189579:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6465 00000100:00080000:2.0:1788575317.189583:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575317.189589:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575317.189595:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575317.189611:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488882432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575317.189612:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575317/real 1788575317] req@ffff94d408c60000 x1875450488882048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575317.189644:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575317.189649:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1788575317.189650:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488882048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575317.189652:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575317.189663:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6465 00000100:00080000:0.0:1788575317.189668:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575317.189670:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575317.189674:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:2.0:1788575317.189679:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575317.189680:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488882048:192.168.202.125@tcp:250:kworker.0 00000008:00100000:2.0:1788575317.189689:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575317.189704:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488882560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575317.189712:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488882176:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575317.189738:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575317.189749:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575317/real 1788575317] req@ffff94d408c62d80 x1875450488882176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575317.189759:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488882176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575317.189766:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575317.189770:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575317.189774:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488882176:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575317.189780:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488882304:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575317.189802:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575317.189814:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575317/real 1788575317] req@ffff94d408c61180 x1875450488882304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575317.189824:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488882304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575317.189836:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575317.189838:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575317.189840:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488882304:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575317.189845:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488882432:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575317.189860:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575317.189870:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575317/real 1788575317] req@ffff94d408c63480 x1875450488882432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575317.189882:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488882432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575317.189890:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575317.189893:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575317.189897:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488882432:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575317.189913:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488882560:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575317.189934:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575317.189945:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575317/real 1788575317] req@ffff94d408c61f80 x1875450488882560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575317.189956:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488882560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575317.189964:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575317.189967:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575317.189970:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488882560:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575317.701386:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575318.149472:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575318.725347:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575319.173424:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575319.749368:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575320.197354:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575320.773367:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575321.221426:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575321.797315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575322.245353:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575322.309328:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575322.309339:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575322.309344:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575322.309355:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6470 00000100:00080000:2.0:1788575322.309365:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575322.309373:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575322.309411:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488882688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575322.309435:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575322.309438:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575322.309440:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575322.309444:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6470 00000100:00080000:2.0:1788575322.309447:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575322.309450:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575322.309459:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575322.309476:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488882816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575322.309491:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575322.309493:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488882688:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788575322.309497:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575322.309508:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575322.309515:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6470 00000100:00080000:2.0:1788575322.309519:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575322.309525:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575322.309531:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575322.309555:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488882944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575322.309563:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575322.309565:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575322.309567:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575322.309570:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6470 00000100:00080000:2.0:1788575322.309574:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575322.309577:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575322.309581:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788575322.309586:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575322.309592:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488883072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575322.309600:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575322.309603:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575322.309606:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575322.309610:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6470 00000100:00080000:2.0:1788575322.309615:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575322.309620:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575322.309626:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575322.309642:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488883200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575322.309654:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575322/real 1788575322] req@ffff94d408c62a00 x1875450488882688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575322.309684:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488882688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575322.309698:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575322.309703:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575322.309706:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488882688:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575322.309735:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488882816:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575322.309758:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575322.309775:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575322/real 1788575322] req@ffff94d408c63b80 x1875450488882816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575322.309793:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488882816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575322.309803:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575322.309810:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575322.309814:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488882816:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575322.309824:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488882944:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575322.309853:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575322.309884:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575322/real 1788575322] req@ffff94d408c60a80 x1875450488882944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575322.309900:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488882944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575322.309917:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575322.309922:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575322.309925:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488882944:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575322.309934:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488883072:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575322.309960:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575322.309974:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575322/real 1788575322] req@ffff94d408c61500 x1875450488883072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575322.309988:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488883072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575322.309997:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575322.310001:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575322.310004:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488883072:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575322.310012:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488883200:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575322.310036:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575322.310049:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575322/real 1788575322] req@ffff94d408c61880 x1875450488883200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575322.310062:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488883200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575322.310071:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575322.310075:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575322.310079:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488883200:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575322.821326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575323.269383:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575323.845321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575324.293340:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575324.869318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575325.317364:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575325.893296:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575326.341327:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575326.917308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575327.365371:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575327.429328:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575327.429336:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575327.429340:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575327.429348:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6475 00000100:00080000:1.0:1788575327.429357:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575327.429363:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575327.429395:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd880 x1875450488883328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575327.429419:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575327.429422:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575327.429423:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575327.429426:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6475 00000100:00080000:1.0:1788575327.429428:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575327.429431:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575327.429438:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575327.429446:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488883456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575327.429455:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575327.429457:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575327.429459:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575327.429463:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488883328:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788575327.429464:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6475 00000100:00080000:1.0:1788575327.429469:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575327.429473:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575327.429477:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575327.429487:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488883584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575327.429494:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575327.429496:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575327.429497:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575327.429500:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6475 00000100:00080000:1.0:1788575327.429510:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575327.429513:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575327.429516:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788575327.429523:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575327.429526:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488883712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575327.429534:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575327.429537:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575327.429538:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575327.429541:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6475 00000100:00080000:1.0:1788575327.429544:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575327.429546:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575327.429550:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575327.429558:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488883840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575327.429562:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575327/real 1788575327] req@ffff94d4049bd880 x1875450488883328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575327.429592:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd880 x1875450488883328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575327.429605:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575327.429610:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575327.429615:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488883328:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575327.429637:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488883456:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575327.429657:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575327.429669:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575327/real 1788575327] req@ffff94d4049bca80 x1875450488883456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575327.429682:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488883456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575327.429691:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575327.429695:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575327.429699:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488883456:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575327.429705:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488883584:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575327.429725:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575327.429736:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575327/real 1788575327] req@ffff94d4049be300 x1875450488883584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575327.429749:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488883584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575327.429758:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575327.429762:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575327.429765:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488883584:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575327.429772:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488883712:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575327.429791:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575327.429800:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575327/real 1788575327] req@ffff94d4049bf100 x1875450488883712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575327.429808:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488883712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575327.429814:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575327.429816:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575327.429819:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488883712:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575327.429822:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488883840:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575327.429834:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575327.429839:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575327/real 1788575327] req@ffff94d4049be680 x1875450488883840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575327.429847:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488883840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575327.429851:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575327.429853:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575327.429855:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488883840:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575327.941263:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575328.389381:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575328.965289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575329.413349:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575329.989273:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575330.437375:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575331.013289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575331.461360:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575332.037300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575332.485384:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575332.549340:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575332.549349:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575332.549352:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575332.549360:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6480 00000100:00080000:2.0:1788575332.549369:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575332.549375:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575332.549406:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488883968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575332.549432:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575332.549435:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575332.549436:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575332.549438:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6480 00000100:00080000:2.0:1788575332.549441:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575332.549443:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575332.549451:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575332.549459:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60e00 x1875450488884096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575332.549467:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575332.549469:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575332.549470:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575332.549473:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6480 00000100:00080000:2.0:1788575332.549475:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575332.549477:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575332.549482:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488883968:192.168.202.125@tcp:250:kworker.0 00000008:00100000:2.0:1788575332.549483:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575332.549494:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63100 x1875450488884224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575332.549502:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575332.549504:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575332.549505:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575332.549508:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6480 00000100:00080000:2.0:1788575332.549511:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575332.549513:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575332.549517:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575332.549525:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488884352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575332.549531:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575332.549533:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575332.549534:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575332.549536:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6480 00000100:00080000:2.0:1788575332.549539:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788575332.549542:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788575332.549545:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575332.549550:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575332.549558:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488884480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575332.549581:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575332/real 1788575332] req@ffff94d408c61c00 x1875450488883968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575332.549598:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488883968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575332.549609:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575332.549613:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575332.549618:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488883968:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575332.549626:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488884096:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575332.549650:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575332.549660:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575332/real 1788575332] req@ffff94d408c60e00 x1875450488884096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575332.549669:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60e00 x1875450488884096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575332.549674:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575332.549677:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575332.549679:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488884096:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575332.549698:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488884224:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575332.549710:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575332.549717:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575332/real 1788575332] req@ffff94d408c63100 x1875450488884224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575332.549729:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63100 x1875450488884224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575332.549734:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575332.549736:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575332.549738:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488884224:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575332.549741:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488884352:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575332.549752:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575332.549758:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575332/real 1788575332] req@ffff94d408c61880 x1875450488884352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575332.549766:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488884352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575332.549771:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575332.549772:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575332.549774:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488884352:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575332.549778:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488884480:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575332.549787:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575332.549792:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575332/real 1788575332] req@ffff94d408c61500 x1875450488884480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575332.549799:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488884480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575332.549808:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575332.549810:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575332.549812:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488884480:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575333.061282:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575333.509365:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575334.085354:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575334.533402:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575335.109336:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575335.557394:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575336.133318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575336.581374:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575337.157340:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575337.605350:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575337.669309:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575337.669320:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575337.669323:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575337.669333:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6485 00000100:00080000:0.0:1788575337.669344:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575337.669352:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575337.669391:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488884608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575337.669425:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575337.669429:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575337.669432:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575337.669436:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6485 00000100:00080000:0.0:1788575337.669440:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575337.669444:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575337.669455:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575337.669472:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488884736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575337.669485:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575337.669490:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575337.669491:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575337.669496:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6485 00000100:00080000:0.0:1788575337.669501:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575337.669507:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488884608:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788575337.669510:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575337.669517:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575337.669540:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488884864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575337.669555:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575337.669559:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575337.669561:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575337.669565:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6485 00000100:00080000:0.0:1788575337.669570:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575337.669575:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575337.669581:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575337.669598:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488884992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575337.669609:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575337.669613:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1788575337.669618:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788575337.669620:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575337.669627:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6485 00000100:00080000:0.0:1788575337.669634:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575337.669639:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575337.669646:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575337.669663:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488885120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575337.669672:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575337/real 1788575337] req@ffff94d408d3ca80 x1875450488884608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575337.669710:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488884608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575337.669722:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575337.669726:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575337.669729:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488884608:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575337.669737:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488884736:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575337.669756:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575337.669795:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575337/real 1788575337] req@ffff94d408d3ea00 x1875450488884736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575337.669813:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488884736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575337.669823:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575337.669826:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575337.669829:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488884736:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575337.669857:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488884864:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575337.669883:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575337.669897:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575337/real 1788575337] req@ffff94d408d3e680 x1875450488884864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575337.669922:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488884864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575337.669928:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575337.669931:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575337.669933:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488884864:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575337.669938:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488884992:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575337.669961:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575337.669970:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575337/real 1788575337] req@ffff94d408d3c000 x1875450488884992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575337.669979:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488884992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575337.669985:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575337.669987:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575337.669989:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488884992:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575337.669993:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488885120:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575337.670005:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575337.670011:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575337/real 1788575337] req@ffff94d408d3d500 x1875450488885120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575337.670018:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488885120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575337.670028:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575337.670030:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575337.670032:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488885120:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575338.181320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575338.629323:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575339.205322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575339.653388:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575340.229427:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575340.677365:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575341.253311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575341.701366:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575342.277315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575342.725367:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575342.789315:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575342.789326:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575342.789330:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575342.789340:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6491 00000100:00080000:0.0:1788575342.789351:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575342.789359:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575342.789397:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f100 x1875450488885248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575342.789422:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575342.789426:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575342.789428:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575342.789431:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6491 00000100:00080000:0.0:1788575342.789434:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575342.789437:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575342.789445:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575342.789455:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f800 x1875450488885376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575342.789463:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575342.789465:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575342.789467:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575342.789473:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6491 00000100:00100000:1.0:1788575342.789473:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488885248:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788575342.789477:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575342.789483:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575342.789487:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575342.789495:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488885504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575342.789501:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575342.789504:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575342.789505:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575342.789508:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6491 00000100:00080000:0.0:1788575342.789510:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575342.789513:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575342.789516:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575342.789527:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488885632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575342.789537:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575342.789541:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575342.789546:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788575342.789546:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788575342.789554:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6491 00000100:00080000:0.0:1788575342.789559:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575342.789565:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575342.789571:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575342.789585:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488885760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575342.789589:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575342/real 1788575342] req@ffff94d408d3f100 x1875450488885248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575342.789622:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f100 x1875450488885248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575342.789632:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575342.789635:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575342.789638:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488885248:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575342.789657:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488885376:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575342.789674:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575342.789683:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575342/real 1788575342] req@ffff94d408d3f800 x1875450488885376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575342.789719:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f800 x1875450488885376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575342.789725:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575342.789728:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575342.789730:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488885376:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575342.789735:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488885504:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575342.789750:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575342.789757:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575342/real 1788575342] req@ffff94d408d3d500 x1875450488885504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575342.789765:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488885504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575342.789771:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575342.789773:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575342.789775:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488885504:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575342.789778:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488885632:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575342.789788:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575342.789793:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575342/real 1788575342] req@ffff94d408d3c000 x1875450488885632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575342.789801:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488885632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575342.789806:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575342.789808:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575342.789809:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488885632:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575342.789813:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488885760:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575342.789822:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575342.789827:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575342/real 1788575342] req@ffff94d408d3e680 x1875450488885760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575342.789835:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488885760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575342.789840:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575342.789841:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575342.789843:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488885760:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575343.301324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575343.749356:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575344.325310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575344.773342:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575345.349306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575345.797372:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575346.373302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575346.821349:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575347.397380:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575347.845340:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575347.909327:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575347.909338:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575347.909343:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575347.909354:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6496 00000100:00080000:1.0:1788575347.909366:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575347.909374:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575347.909420:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488885888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575347.909452:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575347.909455:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575347.909457:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575347.909460:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6496 00000100:00080000:1.0:1788575347.909462:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575347.909465:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575347.909475:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575347.909485:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488886016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575347.909498:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575347.909501:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575347.909503:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575347.909506:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6496 00000100:00080000:1.0:1788575347.909508:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575347.909515:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575347.909516:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488885888:192.168.202.125@tcp:250:kworker.0 00000008:00100000:1.0:1788575347.909519:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575347.909531:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488886144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575347.909538:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575347.909541:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575347.909542:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575347.909545:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6496 00000100:00080000:1.0:1788575347.909546:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575347.909549:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575347.909552:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575347.909560:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488886272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575347.909567:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575347.909569:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575347.909570:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575347.909573:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6496 00000100:00080000:1.0:1788575347.909576:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575347.909580:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575347.909583:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575347.909590:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488886400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575347.909671:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488886016:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575347.909712:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488886144:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575347.909740:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488886272:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575347.909763:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488886400:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575348.421328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575348.869416:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575349.445354:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575349.893431:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575350.469280:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575350.917351:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788575350.981644:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:1.0:1788575350.981715:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788575350.981740:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575350.981793:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:0.0:1788575350.981804:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575347/real 1788575350] req@ffff94d4049bdc00 x1875450488885888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1788575350.981810:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575350.981826:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575350.981841:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:0.0:1788575350.981846:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488885888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575350.981862:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575350.981868:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575350.981874:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488885888:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575350.981900:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575347/real 1788575350] req@ffff94d4049bd500 x1875450488886016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575350.981915:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488886016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575350.981925:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575350.981929:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575350.981933:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488886016:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575350.981941:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575347/real 1788575350] req@ffff94d4049bf800 x1875450488886144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575350.981954:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488886144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575350.981970:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575350.981990:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575350.981995:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488886144:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575350.982004:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575347/real 1788575350] req@ffff94d4049bed80 x1875450488886272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575350.982026:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488886272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575350.982046:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575350.982051:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575350.982055:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488886272:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575350.982062:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575347/real 1788575350] req@ffff94d4049bc700 x1875450488886400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575350.982075:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488886400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575350.982082:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575350.982085:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575350.982089:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488886400:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575351.493314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575351.941441:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575352.517292:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575352.965384:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575353.029419:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575353.029431:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575353.029436:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575353.029447:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6501 00000100:00080000:2.0:1788575353.029459:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575353.029468:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575353.029507:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488886528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575353.029558:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575353.029564:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575353.029566:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575353.029571:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6501 00000100:00080000:2.0:1788575353.029576:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575353.029590:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575353.029603:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575353.029621:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488886528:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575353.029623:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488886656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575353.029641:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575353.029644:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575353.029645:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575353.029649:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6501 00000100:00080000:2.0:1788575353.029652:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575353.029656:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575353.029661:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575353.029679:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488886784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575353.029692:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575353.029696:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575353.029697:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575353.029700:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6501 00000100:00080000:2.0:1788575353.029702:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575353.029705:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575353.029709:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788575353.029716:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575353.029724:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488886912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575353.029736:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575353.029740:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575353.029743:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575353.029748:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6501 00000100:00080000:2.0:1788575353.029753:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575353.029759:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575353.029766:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575353.029772:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575353/real 1788575353] req@ffff94d408c60a80 x1875450488886528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575353.029788:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488887040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575353.029801:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488886528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575353.029817:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575353.029822:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575353.029828:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488886528:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575353.029866:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488886656:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575353.029890:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575353.029901:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575353/real 1788575353] req@ffff94d408c63b80 x1875450488886656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575353.029911:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488886656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575353.029917:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575353.029920:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575353.029922:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488886656:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575353.029926:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488886784:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575353.029938:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575353.029944:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575353/real 1788575353] req@ffff94d408c62a00 x1875450488886784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575353.029958:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488886784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575353.029964:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575353.029966:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575353.029969:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488886784:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575353.029974:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488886912:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575353.029993:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575353.030005:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575353/real 1788575353] req@ffff94d408c61f80 x1875450488886912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575353.030017:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488886912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575353.030025:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575353.030028:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575353.030031:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488886912:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575353.030037:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488887040:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575353.030059:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575353.030081:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575353/real 1788575353] req@ffff94d408c63480 x1875450488887040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575353.030095:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488887040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575353.030111:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575353.030114:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575353.030118:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488887040:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575353.541353:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575353.989417:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575354.565323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575355.013387:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575355.589311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575356.037421:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575356.613274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575357.061361:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575357.637744:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575358.085408:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575358.149376:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575358.149392:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575358.149398:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575358.149418:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6506 00000100:00080000:1.0:1788575358.149432:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575358.149442:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575358.149499:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488887168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575358.149534:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575358.149538:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575358.149541:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575358.149546:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6506 00000100:00080000:1.0:1788575358.149552:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575358.149558:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575358.149571:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575358.149576:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488887168:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575358.149593:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488887296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575358.149606:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575358.149611:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575358.149613:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575358.149619:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6506 00000100:00080000:1.0:1788575358.149624:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575358.149630:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575358.149636:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575358.149653:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488887424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788575358.149665:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788575358.149668:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575358.149672:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575358.149674:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575358.149678:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6506 00000100:00080000:1.0:1788575358.149682:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575358.149686:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575358.149692:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575358.149704:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488887552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575358.149713:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575358/real 1788575358] req@ffff94d4049bc000 x1875450488887168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575358.149718:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575358.149727:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575358.149730:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575358.149736:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6506 00000100:00080000:1.0:1788575358.149739:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575358.149742:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575358.149746:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575358.149753:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488887168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575358.149756:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488887680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575358.149790:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575358.149807:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575358.149815:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488887168:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575358.149842:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488887296:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575358.149866:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575358.149879:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575358/real 1788575358] req@ffff94d4049bfb80 x1875450488887296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575358.149892:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488887296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575358.149901:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575358.149905:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575358.149909:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488887296:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575358.149915:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488887424:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575358.149954:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575358.149969:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575358/real 1788575358] req@ffff94d4049bd180 x1875450488887424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575358.149982:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488887424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575358.149997:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575358.150001:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575358.150005:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488887424:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575358.150011:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488887552:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575358.150034:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575358.150047:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575358/real 1788575358] req@ffff94d4049bce00 x1875450488887552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575358.150059:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488887552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575358.150069:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575358.150087:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575358.150094:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488887552:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575358.150105:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488887680:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575358.150148:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575358.150167:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575358/real 1788575358] req@ffff94d4049bea00 x1875450488887680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575358.150185:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488887680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575358.150195:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575358.150198:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575358.150240:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488887680:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575358.661479:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575359.109417:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575359.685410:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575360.133467:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575360.709393:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575361.157398:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575361.733509:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575362.181465:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575362.757276:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575363.205319:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575363.269286:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575363.269297:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575363.269302:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575363.269313:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6511 00000100:00080000:1.0:1788575363.269325:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575363.269333:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575363.269377:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488887808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575363.269427:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575363.269431:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575363.269432:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575363.269435:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6511 00000100:00080000:1.0:1788575363.269437:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575363.269440:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575363.269453:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575363.269464:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488887936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575363.269473:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575363.269476:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575363.269478:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575363.269484:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6511 00000100:00100000:2.0:1788575363.269487:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488887808:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788575363.269488:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575363.269500:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575363.269507:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575363.269518:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488888064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575363.269527:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575363.269531:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575363.269533:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575363.269537:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6511 00000100:00080000:1.0:1788575363.269540:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575363.269543:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575363.269548:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575363.269557:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488888192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575363.269568:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575363.269571:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575363.269573:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575363.269576:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6511 00000800:00000100:2.0:1788575363.269586:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788575363.269589:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575363.269595:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575363.269599:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575363.269608:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488888320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575363.269629:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575363/real 1788575363] req@ffff94d4049bc380 x1875450488887808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575363.269645:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc380 x1875450488887808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575363.269655:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575363.269658:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575363.269661:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488887808:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575363.269686:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488887936:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575363.269712:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575363.269726:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575363/real 1788575363] req@ffff94d4049bf480 x1875450488887936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575363.269738:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488887936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575363.269744:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575363.269747:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575363.269750:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488887936:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575363.269754:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488888064:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575363.269769:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575363.269776:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575363/real 1788575363] req@ffff94d4049bea00 x1875450488888064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575363.269784:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488888064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575363.269789:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575363.269791:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575363.269793:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488888064:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575363.269797:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488888192:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575363.269808:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575363.269813:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575363/real 1788575363] req@ffff94d4049bce00 x1875450488888192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575363.269821:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488888192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575363.269826:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575363.269827:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575363.269829:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488888192:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575363.269833:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488888320:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575363.269842:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575363.269848:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575363/real 1788575363] req@ffff94d4049bd180 x1875450488888320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575363.269855:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488888320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575363.269860:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575363.269861:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575363.269863:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488888320:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575363.781318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575364.229377:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575364.805280:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575365.253403:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575365.829360:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575366.277406:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575366.853278:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575367.301442:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575367.877289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575368.325388:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575368.389404:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575368.389416:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575368.389421:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575368.389432:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6516 00000100:00080000:0.0:1788575368.389446:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575368.389456:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575368.389510:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488888448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575368.389548:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575368.389552:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575368.389554:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575368.389558:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6516 00000100:00080000:0.0:1788575368.389561:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575368.389565:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575368.389576:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575368.389588:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488888576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575368.389598:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575368.389603:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788575368.389603:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488888448:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788575368.389605:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575368.389620:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6516 00000100:00080000:0.0:1788575368.389623:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575368.389626:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575368.389630:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575368.389640:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488888704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575368.389647:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575368.389650:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575368.389651:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575368.389654:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6516 00000100:00080000:0.0:1788575368.389657:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575368.389660:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575368.389664:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575368.389671:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488888832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575368.389682:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575368.389685:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575368.389686:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788575368.389692:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788575368.389694:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6516 00000100:00080000:0.0:1788575368.389698:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575368.389701:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575368.389705:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575368.389714:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488888960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575368.389743:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575368/real 1788575368] req@ffff94d408d3ea00 x1875450488888448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575368.389761:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488888448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575368.389770:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575368.389773:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575368.389777:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488888448:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575368.389802:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488888576:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575368.389827:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575368.389842:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575368/real 1788575368] req@ffff94d408d3ca80 x1875450488888576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575368.389857:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488888576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575368.389864:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575368.389866:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575368.389869:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488888576:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575368.389874:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488888704:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575368.389889:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575368.389897:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575368/real 1788575368] req@ffff94d408d3d180 x1875450488888704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575368.389906:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488888704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575368.389911:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575368.389914:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575368.389916:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488888704:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575368.389919:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488888832:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575368.389930:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575368.389936:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575368/real 1788575368] req@ffff94d408d3df80 x1875450488888832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575368.389943:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488888832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575368.389949:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575368.389952:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575368.389954:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488888832:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575368.389957:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488888960:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575368.389967:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575368.389972:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575368/real 1788575368] req@ffff94d408d3ce00 x1875450488888960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575368.389978:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488888960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575368.389983:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575368.389986:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575368.389988:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488888960:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575368.901317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575369.349435:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575369.925325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575370.373438:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575370.949384:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575371.397456:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575371.973315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575372.421467:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575372.997354:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575373.445437:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575373.509329:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575373.509339:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575373.509344:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575373.509357:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6521 00000100:00080000:2.0:1788575373.509370:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575373.509379:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575373.509440:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488889088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575373.509472:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575373.509475:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575373.509476:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575373.509480:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6521 00000100:00080000:2.0:1788575373.509484:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575373.509488:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575373.509498:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575373.509507:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488889088:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575373.509513:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488889216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575373.509522:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575373.509525:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575373.509526:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575373.509529:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6521 00000100:00080000:2.0:1788575373.509532:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575373.509534:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575373.509537:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575373.509545:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488889344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575373.509551:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575373.509553:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575373.509554:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575373.509558:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6521 00000100:00080000:2.0:1788575373.509561:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575373.509564:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575373.509568:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575373.509575:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488889472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575373.509581:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575373.509585:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1788575373.509587:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788575373.509588:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575373.509595:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6521 00000100:00080000:2.0:1788575373.509597:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575373.509600:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575373.509604:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575373.509612:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488889600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575373.509633:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575373/real 1788575373] req@ffff94d408c61180 x1875450488889088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575373.509651:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488889088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575373.509660:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575373.509663:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575373.509666:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488889088:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575373.509688:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488889216:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575373.509705:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575373.509714:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575373/real 1788575373] req@ffff94d408c62d80 x1875450488889216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575373.509723:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488889216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575373.509728:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575373.509731:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575373.509733:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488889216:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575373.509737:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488889344:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575373.509747:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575373.509752:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575373/real 1788575373] req@ffff94d408c60000 x1875450488889344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575373.509759:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488889344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575373.509765:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575373.509767:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575373.509769:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488889344:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575373.509772:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488889472:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575373.509783:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575373.509788:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575373/real 1788575373] req@ffff94d408c63800 x1875450488889472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575373.509800:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488889472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575373.509806:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575373.509807:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575373.509809:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488889472:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575373.509813:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488889600:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575373.509822:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575373.509827:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575373/real 1788575373] req@ffff94d408c62680 x1875450488889600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575373.509835:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488889600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575373.509840:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575373.509842:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575373.509843:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488889600:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575374.021414:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575374.469403:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575375.045336:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575375.493461:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575376.069364:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575376.517432:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575377.093370:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575377.541455:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575378.117381:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575378.565393:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575378.629327:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575378.629337:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575378.629342:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575378.629355:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6526 00000100:00080000:2.0:1788575378.629368:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575378.629376:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575378.629420:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62300 x1875450488889728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575378.629447:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575378.629450:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575378.629451:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575378.629455:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6526 00000100:00080000:2.0:1788575378.629457:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575378.629460:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575378.629468:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575378.629477:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60380 x1875450488889856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575378.629484:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575378.629487:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575378.629489:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575378.629492:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6526 00000100:00080000:2.0:1788575378.629494:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575378.629496:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575378.629499:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575378.629510:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488889728:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575378.629511:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60700 x1875450488889984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575378.629528:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575378.629532:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575378.629535:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575378.629540:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6526 00000100:00080000:2.0:1788575378.629545:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575378.629550:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575378.629557:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575378.629570:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488890112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575378.629578:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575378.629581:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575378.629583:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575378.629586:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6526 00000100:00080000:2.0:1788575378.629589:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575378.629591:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575378.629595:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788575378.629599:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575378.629607:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488890240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575378.629646:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575378/real 1788575378] req@ffff94d408c62300 x1875450488889728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575378.629668:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62300 x1875450488889728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575378.629682:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575378.629687:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575378.629692:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488889728:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575378.629700:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488889856:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575378.629720:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575378.629731:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575378/real 1788575378] req@ffff94d408c60380 x1875450488889856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575378.629744:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60380 x1875450488889856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575378.629753:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575378.629756:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575378.629760:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488889856:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575378.629797:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488889984:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575378.629816:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575378.629827:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575378/real 1788575378] req@ffff94d408c60700 x1875450488889984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575378.629840:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60700 x1875450488889984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575378.629849:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575378.629860:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575378.629864:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488889984:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575378.629871:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488890112:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575378.629889:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575378.629899:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575378/real 1788575378] req@ffff94d408c62680 x1875450488890112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575378.629919:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488890112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575378.629928:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575378.629932:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575378.629935:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488890112:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575378.629941:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488890240:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575378.629957:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575378.629966:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575378/real 1788575378] req@ffff94d408c63800 x1875450488890240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575378.629978:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488890240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575378.629987:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575378.629990:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575378.629993:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488890240:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575379.141355:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575379.589432:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575380.165375:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575380.613345:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575381.189389:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575381.637440:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575382.213314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575382.661398:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575383.237370:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575383.685420:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575383.749372:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575383.749385:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575383.749389:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575383.749401:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6532 00000100:00080000:3.0:1788575383.749413:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575383.749421:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575383.749467:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488890368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575383.749494:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575383.749497:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575383.749499:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575383.749502:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6532 00000100:00080000:3.0:1788575383.749504:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575383.749507:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575383.749517:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575383.749528:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488890496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575383.749536:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575383.749539:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575383.749540:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575383.749543:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6532 00000100:00080000:3.0:1788575383.749545:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575383.749550:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575383.749553:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488890368:192.168.202.125@tcp:250:kworker.0 00000008:00100000:3.0:1788575383.749554:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575383.749565:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488890624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575383.749573:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575383.749575:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575383.749576:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575383.749579:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6532 00000100:00080000:3.0:1788575383.749580:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575383.749582:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575383.749586:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575383.749595:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488890752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575383.749601:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575383.749604:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575383.749606:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575383.749609:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6532 00000100:00080000:3.0:1788575383.749611:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575383.749614:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575383.749617:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575383.749623:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488890880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575383.749658:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575383.749699:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575383/real 1788575383] req@ffff94d408c54e00 x1875450488890368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575383.749716:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488890368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575383.749725:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575383.749728:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575383.749731:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488890368:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575383.749737:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488890496:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575383.749755:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575383.749763:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575383/real 1788575383] req@ffff94d408c55c00 x1875450488890496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575383.749772:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488890496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575383.749778:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575383.749779:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575383.749781:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488890496:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575383.749804:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488890624:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575383.749817:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575383.749823:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575383/real 1788575383] req@ffff94d408c57100 x1875450488890624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575383.749843:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488890624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575383.749852:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575383.749856:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575383.749859:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488890624:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575383.749866:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488890752:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575383.749888:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575383.749897:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575383/real 1788575383] req@ffff94d408c54a80 x1875450488890752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575383.749905:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488890752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575383.749911:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575383.749913:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575383.749915:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488890752:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575383.749919:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488890880:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575383.749931:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575383.749937:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575383/real 1788575383] req@ffff94d408c56d80 x1875450488890880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575383.749944:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488890880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575383.749954:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575383.749956:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575383.749958:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488890880:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575384.261285:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575384.709342:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575385.285293:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575385.733428:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575386.309330:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575386.757394:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575387.333350:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575387.781424:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575388.357326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575388.805441:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575388.869353:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575388.869369:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575388.869375:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575388.869390:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6537 00000100:00080000:2.0:1788575388.869406:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575388.869416:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575388.869470:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488891008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575388.869517:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575388.869522:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575388.869524:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575388.869529:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6537 00000100:00080000:2.0:1788575388.869533:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575388.869538:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575388.869549:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575388.869564:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488891136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575388.869576:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575388.869580:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575388.869584:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575388.869589:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6537 00000100:00080000:2.0:1788575388.869593:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575388.869598:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575388.869603:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575388.869619:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488891264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575388.869626:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488891008:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788575388.869653:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575388.869659:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575388.869663:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575388.869669:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6537 00000100:00080000:2.0:1788575388.869674:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575388.869679:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575388.869688:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575388.869707:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488891392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575388.869712:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788575388.869727:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575388.869731:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575388.869733:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575388.869739:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6537 00000100:00080000:2.0:1788575388.869745:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575388.869754:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575388.869755:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575388/real 1788575388] req@ffff94d408c60000 x1875450488891008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788575388.869769:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575388.869779:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488891008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575388.869789:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488891520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575388.869792:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575388.869801:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575388.869811:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488891008:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575388.869828:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488891136:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575388.869847:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575388.869863:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575388/real 1788575388] req@ffff94d408c62d80 x1875450488891136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575388.869872:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488891136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575388.869877:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575388.869880:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575388.869882:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488891136:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575388.869904:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488891264:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575388.869915:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575388.869922:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575388/real 1788575388] req@ffff94d408c61180 x1875450488891264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575388.869930:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488891264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575388.869935:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575388.869937:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575388.869939:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488891264:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575388.869942:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488891392:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575388.869953:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575388.869959:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575388/real 1788575388] req@ffff94d408c63480 x1875450488891392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575388.869966:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488891392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575388.869971:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575388.869973:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575388.869975:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488891392:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575388.869978:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488891520:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575388.869988:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575388.870027:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575388/real 1788575388] req@ffff94d408c61f80 x1875450488891520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575388.870036:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488891520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575388.870041:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575388.870044:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575388.870046:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488891520:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575389.381321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575389.829376:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575390.405405:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575390.853399:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575391.429361:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575391.877431:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575392.453355:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575392.901475:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575393.477323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575393.925434:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575393.989357:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575393.989370:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575393.989374:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575393.989386:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6542 00000100:00080000:2.0:1788575393.989399:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575393.989412:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575393.989465:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488891648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575393.989499:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575393.989502:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575393.989503:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575393.989506:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6542 00000100:00080000:2.0:1788575393.989510:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575393.989514:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575393.989525:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575393.989542:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488891776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575393.989550:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575393.989553:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575393.989556:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575393.989560:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6542 00000100:00080000:2.0:1788575393.989564:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575393.989568:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575393.989575:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575393.989585:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488891904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575393.989599:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488891648:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788575393.989606:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575393.989610:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575393.989612:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575393.989621:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6542 00000100:00080000:2.0:1788575393.989625:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575393.989631:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575393.989638:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575393.989650:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488892032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575393.989661:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575393.989665:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575393.989668:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575393.989673:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6542 00000100:00080000:2.0:1788575393.989678:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575393.989683:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575393.989695:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575393.989708:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488892160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575393.989947:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575393.989989:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575393/real 1788575393] req@ffff94d408c62a00 x1875450488891648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575393.990010:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488891648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575393.990022:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575393.990027:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575393.990031:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488891648:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575393.990041:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488891776:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575393.990065:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575393.990078:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575393/real 1788575393] req@ffff94d408c63b80 x1875450488891776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575393.990098:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488891776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575393.990106:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575393.990110:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575393.990113:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488891776:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575393.990137:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488891904:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575393.990158:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575393.990168:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575393/real 1788575393] req@ffff94d408c60a80 x1875450488891904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575393.990181:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488891904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575393.990189:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575393.990192:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575393.990196:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488891904:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575393.990236:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488892032:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575393.990262:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575393.990279:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575393/real 1788575393] req@ffff94d408c61500 x1875450488892032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575393.990306:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488892032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575393.990322:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575393.990325:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575393.990329:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488892032:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575393.990336:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488892160:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575393.990355:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575393.990366:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575393/real 1788575393] req@ffff94d408c61880 x1875450488892160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575393.990378:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488892160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575393.990385:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575393.990389:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575393.990392:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488892160:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575394.501295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575394.949425:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575395.525305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575395.973416:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575396.549304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575396.997494:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575397.573313:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575398.021385:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575398.597322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575399.045377:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575399.109389:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575399.109403:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575399.109409:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575399.109422:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6547 00000100:00080000:0.0:1788575399.109436:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575399.109446:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575399.109495:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488892288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575399.109528:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575399.109531:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575399.109533:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575399.109537:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6547 00000100:00080000:0.0:1788575399.109542:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575399.109547:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575399.109558:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575399.109572:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488892416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575399.109583:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575399.109590:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788575399.109591:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488892288:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788575399.109593:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575399.109602:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6547 00000100:00080000:0.0:1788575399.109607:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575399.109611:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575399.109615:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575399.109627:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488892544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575399.109637:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575399.109647:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575399.109649:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575399.109654:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6547 00000100:00080000:0.0:1788575399.109658:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575399.109663:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575399.109670:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575399.109682:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488892672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575399.109694:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788575399.109695:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575399.109702:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575399.109704:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575399.109710:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6547 00000100:00080000:0.0:1788575399.109715:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575399.109719:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575399.109725:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575399.109738:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488892800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575399.109749:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575399/real 1788575399] req@ffff94d408d3f480 x1875450488892288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575399.109774:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488892288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575399.109787:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575399.109792:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575399.109797:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488892288:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575399.109823:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488892416:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575399.109847:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575399.109857:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575399/real 1788575399] req@ffff94d408d3c700 x1875450488892416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575399.109894:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488892416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575399.109900:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575399.109903:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575399.109905:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488892416:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575399.109910:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488892544:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575399.109925:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575399.109933:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575399/real 1788575399] req@ffff94d408d3e300 x1875450488892544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575399.109941:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488892544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575399.109946:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575399.109948:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575399.109950:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488892544:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575399.109954:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488892672:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575399.109966:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575399.109971:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575399/real 1788575399] req@ffff94d408d3dc00 x1875450488892672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575399.109979:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488892672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575399.109984:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575399.109986:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575399.109988:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488892672:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575399.109993:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488892800:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575399.110010:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575399.110023:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575399/real 1788575399] req@ffff94d408d3c380 x1875450488892800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575399.110036:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488892800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575399.110044:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575399.110048:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575399.110051:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488892800:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575399.621282:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575400.069366:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575400.645359:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575401.093366:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575401.669283:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575402.117396:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575402.693312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575403.141403:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575403.717328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575404.165426:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575404.229364:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575404.229376:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575404.229380:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575404.229391:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6552 00000100:00080000:2.0:1788575404.229413:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575404.229422:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575404.229471:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63100 x1875450488892928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575404.229495:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575404.229498:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575404.229500:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575404.229502:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6552 00000100:00080000:2.0:1788575404.229505:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575404.229508:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575404.229516:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575404.229525:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60e00 x1875450488893056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575404.229532:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575404.229535:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575404.229536:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575404.229539:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6552 00000100:00080000:2.0:1788575404.229541:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575404.229543:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575404.229546:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575404.229550:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488892928:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575404.229555:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488893184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575404.229563:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575404.229564:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575404.229565:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575404.229568:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6552 00000100:00080000:2.0:1788575404.229569:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575404.229571:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575404.229574:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575404.229583:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488893312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575404.229588:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575404.229590:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575404.229591:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575404.229593:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6552 00000100:00080000:2.0:1788575404.229595:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575404.229598:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575404.229601:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575404.229608:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488893440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575404.229634:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575404.229686:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575404/real 1788575404] req@ffff94d408c63100 x1875450488892928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575404.229715:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63100 x1875450488892928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575404.229728:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575404.229741:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575404.229745:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488892928:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575404.229752:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488893056:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575404.229773:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575404.229786:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575404/real 1788575404] req@ffff94d408c60e00 x1875450488893056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575404.229798:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60e00 x1875450488893056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575404.229807:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575404.229824:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575404.229828:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488893056:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575404.229857:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488893184:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575404.229878:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575404.229889:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575404/real 1788575404] req@ffff94d408c61c00 x1875450488893184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575404.229901:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488893184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575404.229916:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575404.229919:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575404.229922:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488893184:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575404.229927:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488893312:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575404.229946:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575404.229957:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575404/real 1788575404] req@ffff94d408c61880 x1875450488893312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575404.229968:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488893312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575404.229976:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575404.229979:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575404.229982:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488893312:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575404.229988:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488893440:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575404.230004:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575404.230014:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575404/real 1788575404] req@ffff94d408c61500 x1875450488893440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575404.230026:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488893440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575404.230033:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575404.230036:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575404.230040:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488893440:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575404.741373:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575405.189418:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575405.765284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575406.213404:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575406.789330:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575407.237414:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575407.813364:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575408.261458:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575408.837361:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575409.285412:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575409.349377:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575409.349385:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575409.349388:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575409.349395:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6557 00000100:00080000:1.0:1788575409.349403:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575409.349409:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575409.349439:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488893568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575409.349463:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575409.349465:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575409.349466:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575409.349468:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6557 00000100:00080000:1.0:1788575409.349470:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575409.349472:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575409.349479:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575409.349486:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488893696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575409.349505:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575409.349508:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575409.349511:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788575409.349514:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488893568:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788575409.349518:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6557 00000100:00080000:1.0:1788575409.349525:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575409.349529:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575409.349534:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575409.349547:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488893824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575409.349556:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575409.349560:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575409.349562:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575409.349566:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6557 00000100:00080000:1.0:1788575409.349570:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575409.349574:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575409.349579:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788575409.349586:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575409.349593:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488893952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575409.349603:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575409.349607:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575409.349608:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575409.349612:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6557 00000100:00080000:1.0:1788575409.349616:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575409.349620:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575409.349625:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575409/real 1788575409] req@ffff94d4049bfb80 x1875450488893568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788575409.349627:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575409.349639:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488894080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575409.349649:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488893568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575409.349660:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575409.349664:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575409.349668:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488893568:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575409.349692:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488893696:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575409.349720:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575409.349731:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575409/real 1788575409] req@ffff94d4049bc000 x1875450488893696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575409.349743:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488893696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575409.349751:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575409.349754:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575409.349757:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488893696:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575409.349774:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488893824:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575409.349793:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575409.349803:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575409/real 1788575409] req@ffff94d4049bc700 x1875450488893824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575409.349814:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488893824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575409.349822:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575409.349825:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575409.349828:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488893824:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575409.349834:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488893952:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575409.349852:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575409.349871:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575409/real 1788575409] req@ffff94d4049bed80 x1875450488893952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575409.349881:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488893952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575409.349889:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575409.349892:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575409.349895:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488893952:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575409.349901:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488894080:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575409.349919:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575409.349928:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575409/real 1788575409] req@ffff94d4049bf800 x1875450488894080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575409.349945:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488894080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575409.349953:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575409.349956:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575409.349959:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488894080:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575409.861359:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575410.309449:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575410.885310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575411.333440:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575411.909294:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575412.357361:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575412.933307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575413.381320:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575413.957314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575414.405386:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575414.469283:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575414.469290:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575414.469293:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575414.469301:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6562 00000100:00080000:0.0:1788575414.469308:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575414.469314:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575414.469347:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ed80 x1875450488894208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575414.469370:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575414.469373:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575414.469374:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575414.469377:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6562 00000100:00080000:0.0:1788575414.469379:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575414.469382:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575414.469389:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575414.469399:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488894336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575414.469407:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488894208:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788575414.469408:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575414.469415:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575414.469417:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575414.469420:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6562 00000100:00080000:0.0:1788575414.469422:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575414.469424:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575414.469427:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575414.469435:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488894464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575414.469441:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575414.469444:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575414.469446:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575414.469449:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6562 00000100:00080000:0.0:1788575414.469453:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575414.469457:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575414.469467:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575414.469480:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488894592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575414.469497:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488894336:192.168.202.125@tcp:38:kworker.0 00000100:00080000:0.0:1788575414.469500:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575414.469504:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575414.469507:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575414.469511:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6562 00000100:00080000:0.0:1788575414.469515:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575414.469520:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575414.469526:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575414.469530:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488894464:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575414.469540:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488894720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575414.469554:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488894592:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575414.469570:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488894720:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575414.981280:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575415.429342:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575416.005302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575416.453352:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575417.029292:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575417.477390:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788575417.541412:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:1.0:1788575417.541452:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788575417.541467:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575417.541503:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575417.541513:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575417.541521:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575417.541529:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:3.0:1788575417.541551:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575414/real 1788575417] req@ffff94d408d3dc00 x1875450488894592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575417.541575:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488894592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575417.541586:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575417.541597:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575417.541601:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488894592:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575417.541610:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575414/real 1788575417] req@ffff94d408d3e300 x1875450488894720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575417.541621:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488894720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575417.541628:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575417.541631:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575417.541634:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488894720:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575417.541640:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575414/real 1788575417] req@ffff94d408d3d880 x1875450488894336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575417.541652:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488894336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575417.541657:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575417.541660:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575417.541662:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488894336:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575417.541667:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575414/real 1788575417] req@ffff94d408d3c380 x1875450488894464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575417.541677:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488894464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575417.541685:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575417.541702:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575417.541705:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488894464:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575417.541712:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575414/real 1788575417] req@ffff94d408d3ed80 x1875450488894208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575417.541726:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ed80 x1875450488894208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575417.541733:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575417.541736:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575417.541739:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488894208:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575418.053332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575418.501381:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575419.077293:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575419.525367:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575419.589284:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575419.589297:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575419.589301:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575419.589311:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6567 00000100:00080000:0.0:1788575419.589323:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575419.589331:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575419.589371:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488894848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575419.589404:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575419.589408:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575419.589410:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575419.589414:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6567 00000100:00080000:0.0:1788575419.589417:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575419.589420:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575419.589431:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575419.589448:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488894976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575419.589449:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488894848:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788575419.589465:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575419.589469:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575419.589470:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575419.589474:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6567 00000100:00080000:0.0:1788575419.589478:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575419.589482:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575419.589487:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575419.589502:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488895104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575419.589511:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575419.589515:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575419.589517:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575419.589522:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6567 00000100:00080000:0.0:1788575419.589525:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575419.589528:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575419.589532:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575419.589546:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488895232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575419.589556:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788575419.589558:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575419.589562:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575419.589564:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575419.589569:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6567 00000100:00080000:0.0:1788575419.589573:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575419.589578:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575419.589583:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575419.589595:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488895360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575419.589604:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575419/real 1788575419] req@ffff94d408d3c700 x1875450488894848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575419.589628:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488894848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575419.589641:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575419.589645:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575419.589649:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488894848:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575419.589720:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488894976:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575419.589745:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575419.589758:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575419/real 1788575419] req@ffff94d408d3f480 x1875450488894976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575419.589770:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488894976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575419.589779:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575419.589782:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575419.589786:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488894976:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575419.589791:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488895104:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575419.589812:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575419.589828:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575419/real 1788575419] req@ffff94d408d3ce00 x1875450488895104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575419.589839:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488895104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575419.589847:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575419.589850:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575419.589853:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488895104:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575419.589859:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488895232:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575419.589879:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575419.589889:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575419/real 1788575419] req@ffff94d408d3df80 x1875450488895232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575419.589900:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488895232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575419.589907:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575419.589910:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575419.589914:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488895232:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575419.589919:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488895360:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575419.589936:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575419.589946:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575419/real 1788575419] req@ffff94d408d3d180 x1875450488895360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575419.589956:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488895360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575419.589963:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575419.589965:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575419.589968:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488895360:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575420.101377:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575420.549428:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575421.125320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575421.573298:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575422.149336:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575422.597382:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575423.174059:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575423.621373:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575424.197937:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575424.645361:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575424.709294:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575424.709303:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575424.709306:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575424.709315:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6573 00000100:00080000:0.0:1788575424.709325:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575424.709331:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575424.709362:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488895488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575424.709382:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575424.709385:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575424.709387:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575424.709389:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6573 00000100:00080000:0.0:1788575424.709392:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575424.709395:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575424.709402:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575424.709410:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488895616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575424.709418:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575424.709423:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788575424.709424:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488895488:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788575424.709425:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575424.709433:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6573 00000100:00080000:0.0:1788575424.709436:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575424.709439:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575424.709442:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575424.709450:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488895744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575424.709457:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575424.709460:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575424.709461:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575424.709464:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6573 00000100:00080000:0.0:1788575424.709466:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575424.709469:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575424.709472:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575424.709479:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488895872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575424.709486:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575424.709488:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575424.709489:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575424.709494:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6573 00000800:00000100:3.0:1788575424.709497:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788575424.709498:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575424.709503:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575424.709506:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575424.709512:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488896000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575424.709543:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575424/real 1788575424] req@ffff94d408d3ca80 x1875450488895488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575424.709563:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488895488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575424.709575:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575424.709579:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575424.709584:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488895488:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575424.709609:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488895616:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575424.709659:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575424.709670:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575424/real 1788575424] req@ffff94d408d3ea00 x1875450488895616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575424.709679:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488895616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575424.709685:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575424.709687:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575424.709689:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488895616:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575424.709693:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488895744:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575424.709705:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575424.709711:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575424/real 1788575424] req@ffff94d408d3e680 x1875450488895744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575424.709723:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488895744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575424.709729:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575424.709731:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575424.709733:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488895744:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575424.709736:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488895872:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575424.709748:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575424.709760:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575424/real 1788575424] req@ffff94d408d3c000 x1875450488895872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575424.709773:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488895872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575424.709778:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575424.709780:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575424.709782:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488895872:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575424.709786:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488896000:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575424.709796:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575424.709802:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575424/real 1788575424] req@ffff94d408d3d500 x1875450488896000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575424.709809:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488896000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575424.709814:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575424.709816:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575424.709818:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488896000:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575425.221827:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575425.669382:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575426.245689:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575426.693386:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575427.270011:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575427.717416:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575428.293795:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575428.741413:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575429.317325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575429.765467:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575429.829354:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575429.829363:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575429.829366:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575429.829376:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6578 00000100:00080000:1.0:1788575429.829385:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575429.829393:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575429.829427:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488896128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575429.829454:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575429.829457:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575429.829459:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575429.829464:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6578 00000100:00080000:1.0:1788575429.829470:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575429.829474:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575429.829481:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575429.829491:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488896256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575429.829498:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575429.829501:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575429.829502:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575429.829504:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6578 00000100:00080000:1.0:1788575429.829507:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575429.829510:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575429.829513:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575429.829521:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488896384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575429.829527:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575429.829529:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575429.829530:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575429.829532:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6578 00000100:00080000:1.0:1788575429.829533:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575429.829535:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575429.829539:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575429.829547:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488896512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575429.829553:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575429.829555:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575429.829556:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575429.829559:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6578 00000100:00080000:1.0:1788575429.829562:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575429.829564:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575429.829568:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575429.829597:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488896640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575429.829636:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488896128:192.168.202.125@tcp:250:kworker.0 00000800:00000100:1.0:1788575429.829693:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575429.829731:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575429/real 1788575429] req@ffff94d4049bd500 x1875450488896128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575429.829750:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488896128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575429.829758:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575429.829761:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575429.829764:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488896128:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575429.829771:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488896256:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575429.829793:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575429.829804:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575429/real 1788575429] req@ffff94d4049bdc00 x1875450488896256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575429.829813:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488896256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575429.829818:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575429.829822:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575429.829827:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488896256:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575429.829835:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488896384:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575429.829860:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575429.829875:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575429/real 1788575429] req@ffff94d4049be680 x1875450488896384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575429.829890:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488896384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575429.829900:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575429.829903:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575429.829907:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488896384:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575429.829915:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488896512:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575429.829936:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575429.829946:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575429/real 1788575429] req@ffff94d4049bf100 x1875450488896512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575429.829959:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488896512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575429.829967:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575429.829970:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575429.829974:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488896512:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575429.829982:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488896640:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575429.830007:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575429.830016:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575429/real 1788575429] req@ffff94d4049be300 x1875450488896640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575429.830023:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488896640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575429.830029:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575429.830031:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575429.830033:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488896640:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575430.341305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575430.789318:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575431.365340:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575431.813367:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575432.389311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575432.837355:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575433.413314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575433.861381:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575434.437270:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575434.885356:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575434.949328:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575434.949339:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575434.949343:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575434.949353:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6583 00000100:00080000:0.0:1788575434.949364:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575434.949372:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575434.949412:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f800 x1875450488896768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575434.949439:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575434.949443:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575434.949446:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575434.949450:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6583 00000100:00080000:0.0:1788575434.949455:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575434.949459:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575434.949470:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575434.949470:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488896768:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575434.949488:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f100 x1875450488896896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575434.949501:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575434.949505:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575434.949507:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575434.949512:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6583 00000100:00080000:0.0:1788575434.949536:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575434.949542:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575434.949547:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575434.949555:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575434.949563:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488897024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575434.949576:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575434.949579:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575434.949581:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575434.949586:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6583 00000100:00080000:0.0:1788575434.949590:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575434.949594:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575434/real 1788575434] req@ffff94d408d3f800 x1875450488896768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575434.949597:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575434.949605:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575434.949617:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488897152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575434.949628:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575434.949629:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f800 x1875450488896768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575434.949634:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575434.949638:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575434.949646:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6583 00000100:00080000:0.0:1788575434.949651:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575434.949655:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00080000:1.0:1788575434.949655:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1788575434.949663:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788575434.949666:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575434.949674:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488897280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575434.949683:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488896768:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575434.949712:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488896896:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575434.949740:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575434.949755:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575434/real 1788575434] req@ffff94d408d3f100 x1875450488896896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575434.949770:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f100 x1875450488896896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575434.949779:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575434.949783:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575434.949787:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488896896:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575434.949794:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488897024:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575434.949816:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575434.949836:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575434/real 1788575434] req@ffff94d408d3d500 x1875450488897024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575434.949844:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488897024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575434.949850:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575434.949852:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575434.949854:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488897024:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575434.949859:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488897152:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575434.949873:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575434.949886:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575434/real 1788575434] req@ffff94d408d3c000 x1875450488897152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575434.949894:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488897152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575434.949899:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575434.949901:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575434.949903:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488897152:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575434.949907:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488897280:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575434.949919:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575434.949925:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575434/real 1788575434] req@ffff94d408d3e680 x1875450488897280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575434.949932:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488897280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575434.949937:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575434.949939:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575434.949940:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488897280:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575435.461306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575435.909398:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575436.485332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575436.933429:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575437.509322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575437.957386:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575438.533291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575438.981363:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575439.557282:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575440.005433:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575440.069320:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575440.069329:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575440.069333:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575440.069343:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6588 00000100:00080000:3.0:1788575440.069352:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575440.069359:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575440.069390:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488897408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575440.069413:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575440.069416:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575440.069417:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575440.069420:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6588 00000100:00080000:3.0:1788575440.069422:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575440.069424:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575440.069431:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575440.069439:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488897536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575440.069447:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575440.069450:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575440.069451:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575440.069453:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6588 00000100:00080000:3.0:1788575440.069456:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575440.069458:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575440.069461:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575440.069469:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488897664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575440.069476:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488897408:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575440.069502:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575440.069506:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575440.069508:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575440.069515:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6588 00000100:00080000:3.0:1788575440.069519:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575440.069523:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575440.069529:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575440.069543:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488897792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575440.069553:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575440.069556:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575440.069558:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575440.069561:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6588 00000100:00080000:3.0:1788575440.069565:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788575440.069569:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575440.069573:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575440.069580:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575440.069593:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488897920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575440.069609:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575440/real 1788575440] req@ffff94d408c56680 x1875450488897408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575440.069626:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488897408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575440.069635:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575440.069638:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575440.069643:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488897408:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575440.069651:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488897536:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575440.069676:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575440.069690:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575440/real 1788575440] req@ffff94d408c57b80 x1875450488897536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575440.069704:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488897536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575440.069714:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575440.069717:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575440.069721:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488897536:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575440.069743:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488897664:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575440.069762:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575440.069772:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575440/real 1788575440] req@ffff94d408c56d80 x1875450488897664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575440.069785:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488897664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575440.069794:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575440.069797:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575440.069801:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488897664:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575440.069807:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488897792:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575440.069822:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575440.069831:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575440/real 1788575440] req@ffff94d408c54a80 x1875450488897792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575440.069843:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488897792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575440.069858:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575440.069861:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575440.069864:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488897792:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575440.069869:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488897920:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575440.069884:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575440.069892:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575440/real 1788575440] req@ffff94d408c57100 x1875450488897920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575440.069903:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488897920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575440.069911:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575440.069913:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575440.069915:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488897920:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575440.581357:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575441.029428:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575441.605348:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575442.053519:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575442.629275:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575443.077401:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575443.653308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575444.101366:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575444.677330:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575445.125347:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575445.189320:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575445.189331:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575445.189334:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575445.189341:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6593 00000100:00080000:2.0:1788575445.189349:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575445.189355:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575445.189386:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488898048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575445.189405:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575445.189407:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575445.189409:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575445.189411:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6593 00000100:00080000:2.0:1788575445.189413:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575445.189415:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575445.189421:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575445.189427:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488898176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575445.189447:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575445.189449:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575445.189450:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575445.189453:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6593 00000100:00080000:2.0:1788575445.189455:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575445.189458:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575445.189462:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575445.189469:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488898304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575445.189478:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575445.189482:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575445.189484:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575445.189488:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6593 00000100:00080000:2.0:1788575445.189491:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575445.189495:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575445.189500:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575445.189519:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488898048:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575445.189520:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488898432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575445.189540:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575445.189542:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575445.189544:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575445.189548:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6593 00000100:00080000:2.0:1788575445.189550:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575445.189553:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575445.189557:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575445.189568:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488898560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575445.189618:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575445.189667:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575445/real 1788575445] req@ffff94d408c60a80 x1875450488898048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575445.189682:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488898048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575445.189691:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575445.189695:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575445.189698:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488898048:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575445.189705:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488898176:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575445.189721:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575445.189729:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575445/real 1788575445] req@ffff94d408c63b80 x1875450488898176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575445.189737:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488898176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575445.189742:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575445.189744:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575445.189746:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488898176:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575445.189751:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488898304:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575445.189770:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575445.189781:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575445/real 1788575445] req@ffff94d408c62a00 x1875450488898304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575445.189789:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488898304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575445.189794:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575445.189796:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575445.189798:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488898304:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575445.189824:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488898432:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575445.189837:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575445.189844:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575445/real 1788575445] req@ffff94d408c61f80 x1875450488898432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575445.189851:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488898432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575445.189856:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575445.189858:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575445.189859:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488898432:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575445.189863:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488898560:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575445.189871:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575445.189876:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575445/real 1788575445] req@ffff94d408c63480 x1875450488898560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575445.189882:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488898560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575445.189887:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575445.189893:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575445.189895:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488898560:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575445.701303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575446.149387:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575446.725257:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575447.173356:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575447.749310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575448.197355:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575448.773322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575449.221554:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575449.797301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575450.245444:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575450.309345:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575450.309356:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575450.309360:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575450.309371:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6598 00000100:00080000:1.0:1788575450.309392:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575450.309402:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575450.309452:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488898688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575450.309491:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575450.309495:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575450.309497:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575450.309501:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6598 00000100:00080000:1.0:1788575450.309505:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575450.309509:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575450.309526:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575450.309541:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488898816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575450.309549:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488898688:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788575450.309552:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575450.309558:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575450.309561:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575450.309566:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6598 00000100:00080000:1.0:1788575450.309569:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575450.309572:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575450.309576:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575450.309588:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488898944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575450.309598:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575450.309601:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575450.309604:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575450.309608:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6598 00000100:00080000:1.0:1788575450.309613:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575450.309617:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575450.309623:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575450.309637:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488899072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575450.309637:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788575450.309650:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575450.309653:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575450.309655:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575450.309660:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6598 00000100:00080000:1.0:1788575450.309664:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575450.309668:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575450.309674:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575450.309683:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575450/real 1788575450] req@ffff94d4049be300 x1875450488898688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575450.309688:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488899200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575450.309718:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488898688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575450.309729:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575450.309734:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575450.309738:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488898688:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575450.309763:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488898816:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575450.309785:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575450.309796:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575450/real 1788575450] req@ffff94d4049bf100 x1875450488898816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575450.309807:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488898816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575450.309815:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575450.309818:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575450.309821:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488898816:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575450.309826:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488898944:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575450.309841:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575450.309850:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575450/real 1788575450] req@ffff94d4049be680 x1875450488898944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575450.309866:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488898944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575450.309874:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575450.309877:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575450.309880:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488898944:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575450.309885:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488899072:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575450.309903:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575450.309913:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575450/real 1788575450] req@ffff94d4049bdc00 x1875450488899072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575450.309924:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488899072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575450.309931:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575450.309934:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575450.309937:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488899072:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575450.309942:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488899200:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575450.309957:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575450.309966:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575450/real 1788575450] req@ffff94d4049bd500 x1875450488899200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575450.309977:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488899200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575450.309989:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575450.309992:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575450.309996:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488899200:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575450.821324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575451.269319:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575451.845278:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575452.293425:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575452.869332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575453.317439:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575453.893344:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575454.341422:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575454.917381:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575455.365414:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575455.429357:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575455.429368:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575455.429372:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575455.429382:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6603 00000100:00080000:3.0:1788575455.429393:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575455.429400:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575455.429442:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488899328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575455.429467:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575455.429470:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575455.429471:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575455.429474:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6603 00000100:00080000:3.0:1788575455.429477:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575455.429480:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575455.429488:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575455.429499:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488899328:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575455.429499:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488899456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575455.429513:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575455.429516:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575455.429517:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575455.429519:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6603 00000100:00080000:3.0:1788575455.429522:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575455.429525:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575455.429529:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575455.429542:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488899584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575455.429551:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575455.429555:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575455.429557:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575455.429562:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6603 00000100:00080000:3.0:1788575455.429566:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575455.429570:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:1.0:1788575455.429573:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:3.0:1788575455.429577:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575455.429588:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488899712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575455.429595:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575455.429597:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575455.429598:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575455.429600:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6603 00000100:00080000:3.0:1788575455.429603:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575455.429606:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575455.429610:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575455.429615:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575455/real 1788575455] req@ffff94d408c55c00 x1875450488899328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575455.429619:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488899840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575455.429645:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488899328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575455.429655:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575455.429659:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575455.429662:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488899328:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575455.429684:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488899456:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575455.429698:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575455.429706:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575455/real 1788575455] req@ffff94d408c54e00 x1875450488899456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575455.429714:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488899456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575455.429719:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575455.429721:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575455.429723:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488899456:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575455.429726:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488899584:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575455.429735:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575455.429740:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575455/real 1788575455] req@ffff94d408c56a00 x1875450488899584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575455.429747:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488899584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575455.429752:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575455.429753:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575455.429755:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488899584:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575455.429759:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488899712:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575455.429768:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575455.429773:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575455/real 1788575455] req@ffff94d408c57800 x1875450488899712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575455.429780:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488899712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575455.429785:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575455.429786:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575455.429788:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488899712:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575455.429791:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488899840:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575455.429799:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575455.429803:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575455/real 1788575455] req@ffff94d408c54380 x1875450488899840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575455.429810:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488899840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575455.429814:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575455.429816:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575455.429817:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488899840:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575455.941354:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575456.389402:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575456.965367:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575457.413414:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575457.989284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575458.437388:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575459.013344:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575459.461418:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575460.037385:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575460.485377:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575460.549250:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575460.549258:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575460.549262:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575460.549271:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6608 00000100:00080000:3.0:1788575460.549286:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575460.549293:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575460.549326:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488899968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575460.549364:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575460.549368:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575460.549370:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575460.549374:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6608 00000100:00080000:3.0:1788575460.549377:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575460.549380:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575460.549389:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575460.549402:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488900096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575460.549409:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488899968:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575460.549417:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575460.549421:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575460.549423:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575460.549433:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6608 00000100:00080000:3.0:1788575460.549436:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575460.549440:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575460.549445:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575460.549455:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488900224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575460.549464:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575460.549467:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575460.549469:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788575460.549472:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575460.549476:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6608 00000100:00080000:3.0:1788575460.549481:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575460.549484:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575460.549489:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575460.549506:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575460/real 1788575460] req@ffff94d408c55880 x1875450488899968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575460.549507:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488900352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575460.549536:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575460.549538:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1788575460.549541:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488899968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575460.549542:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575460.549550:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6608 00000100:00080000:1.0:1788575460.549554:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575460.549556:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575460.549561:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:3.0:1788575460.549564:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575460.549568:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488899968:192.168.202.125@tcp:250:kworker.0 00000008:00100000:3.0:1788575460.549571:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575460.549583:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488900480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575460.549588:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488900096:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575460.549610:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575460.549619:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575460/real 1788575460] req@ffff94d408c55180 x1875450488900096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575460.549627:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488900096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575460.549632:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575460.549634:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575460.549636:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488900096:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575460.549640:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488900224:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575460.549650:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575460.549659:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575460/real 1788575460] req@ffff94d408c54700 x1875450488900224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575460.549670:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488900224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575460.549683:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575460.549687:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575460.549690:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488900224:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575460.549696:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488900352:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575460.549712:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575460.549720:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575460/real 1788575460] req@ffff94d408c56300 x1875450488900352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575460.549727:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488900352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575460.549731:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575460.549733:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575460.549735:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488900352:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575460.549745:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488900480:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575460.549754:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575460.549759:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575460/real 1788575460] req@ffff94d408c55f80 x1875450488900480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575460.549768:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488900480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575460.549774:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575460.549777:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575460.549780:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488900480:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575461.061350:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575461.509399:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575462.085261:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575462.533347:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575463.109341:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575463.557602:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575464.133293:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575464.581419:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575465.157323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575465.605391:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575465.669299:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575465.669310:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575465.669316:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575465.669330:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6613 00000100:00080000:3.0:1788575465.669345:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575465.669353:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575465.669398:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488900608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575465.669421:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575465.669424:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575465.669425:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575465.669429:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6613 00000100:00080000:3.0:1788575465.669432:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575465.669435:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575465.669445:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575465.669458:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488900736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575465.669462:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488900608:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575465.669467:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575465.669478:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575465.669481:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575465.669497:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6613 00000100:00080000:3.0:1788575465.669505:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575465.669509:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575465.669513:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575465.669528:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488900864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575465.669540:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575465.669542:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575465.669547:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788575465.669548:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575465.669556:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6613 00000100:00080000:3.0:1788575465.669561:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575465.669585:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575465/real 1788575465] req@ffff94d408c54000 x1875450488900608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575465.669600:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488900608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575465.669607:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575465.669610:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575465.669615:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488900608:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575465.669634:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488900736:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575465.669650:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575465.669659:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575465/real 1788575465] req@ffff94d408c57480 x1875450488900736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575465.669667:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488900736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575465.669671:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575465.669673:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575465.669675:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488900736:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575465.669679:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488900864:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575465.669690:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575465.669695:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575465/real 1788575465] req@ffff94d408c55f80 x1875450488900864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575465.669702:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488900864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575465.669707:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575465.669711:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575465.669713:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488900864:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788575465.669765:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575465.669771:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575465.669816:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488900992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575465.669835:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575465.669838:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575465.669840:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575465.669847:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6613 00000100:00080000:3.0:1788575465.669854:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575465.669857:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575465.669865:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575465.669879:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488901120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575465.669914:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488900992:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575465.669939:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575465.669952:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575465/real 1788575465] req@ffff94d408c56300 x1875450488900992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575465.669972:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488900992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575465.669980:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575465.669992:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575465.669996:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488900992:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575465.670003:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488901120:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575465.670020:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575465.670028:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575465/real 1788575465] req@ffff94d408c54700 x1875450488901120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575465.670035:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488901120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575465.670040:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575465.670042:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575465.670044:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488901120:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575466.181363:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575466.629397:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575467.205295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575467.653324:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575468.229375:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575468.677366:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575469.253321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575469.701357:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575470.277342:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575470.725382:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575470.789326:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575470.789337:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575470.789341:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575470.789351:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6619 00000100:00080000:3.0:1788575470.789363:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575470.789371:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575470.789410:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488901248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575470.789442:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575470.789446:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575470.789448:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575470.789452:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6619 00000100:00080000:3.0:1788575470.789457:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575470.789461:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575470.789471:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575470.789487:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488901376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575470.789489:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488901248:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575470.789507:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575470.789511:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575470.789514:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575470.789519:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6619 00000100:00080000:3.0:1788575470.789524:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575470.789529:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575470.789534:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788575470.789549:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575470.789550:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488901504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575470.789563:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575470.789567:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575470.789569:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575470.789574:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6619 00000100:00080000:3.0:1788575470.789580:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788575470.789586:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575470/real 1788575470] req@ffff94d408c55180 x1875450488901248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575470.789587:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575470.789597:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575470.789606:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488901248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575470.789615:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488901632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575470.789617:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575470.789626:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575470.789633:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488901248:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575470.789652:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575470.789660:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575470.789664:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575470.789670:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6619 00000100:00080000:3.0:1788575470.789675:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575470.789679:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575470.789682:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488901376:192.168.202.125@tcp:38:kworker.0 00000008:00100000:3.0:1788575470.789688:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575470.789706:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488901760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575470.789709:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575470.789722:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575470/real 1788575470] req@ffff94d408c55880 x1875450488901376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575470.789745:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488901376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575470.789751:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575470.789754:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575470.789756:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488901376:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575470.789761:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488901504:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575470.789774:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575470.789780:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575470/real 1788575470] req@ffff94d408c54380 x1875450488901504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575470.789792:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488901504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575470.789800:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575470.789804:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575470.789808:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488901504:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575470.789814:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488901632:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575470.789835:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575470.789848:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575470/real 1788575470] req@ffff94d408c57800 x1875450488901632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575470.789861:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488901632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575470.789869:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575470.789873:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575470.789876:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488901632:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575470.789893:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488901760:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575470.789912:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575470.789923:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575470/real 1788575470] req@ffff94d408c56a00 x1875450488901760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575470.789936:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488901760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575470.789944:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575470.789948:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575470.789952:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488901760:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575471.301301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575471.749354:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575472.325335:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575472.773368:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575473.349325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575473.797405:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575474.373322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575474.821365:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575475.397333:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575475.845426:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575475.909341:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575475.909353:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575475.909357:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575475.909368:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6624 00000100:00080000:3.0:1788575475.909381:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575475.909389:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575475.909433:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488901888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575475.909473:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575475.909478:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575475.909480:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575475.909485:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6624 00000100:00080000:3.0:1788575475.909489:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575475.909493:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575475.909507:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575475.909525:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488902016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575475.909538:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575475.909541:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488901888:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575475.909543:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575475.909548:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575475.909558:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6624 00000100:00080000:3.0:1788575475.909564:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575475.909569:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575475.909575:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575475.909587:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488902144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575475.909595:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575475.909598:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575475.909599:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575475.909603:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6624 00000100:00080000:3.0:1788575475.909606:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575475.909609:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575475.909613:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575475.909621:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488902272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575475.909631:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575475.909635:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575475.909636:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788575475.909638:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575475.909642:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6624 00000100:00080000:3.0:1788575475.909648:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575475.909655:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575475.909661:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575475.909674:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488902400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575475.909699:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575475/real 1788575475] req@ffff94d408c54e00 x1875450488901888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575475.909725:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488901888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575475.909735:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575475.909738:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575475.909742:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488901888:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575475.909767:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488902016:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575475.909814:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575475.909825:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575475/real 1788575475] req@ffff94d408c55c00 x1875450488902016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575475.909835:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488902016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575475.909841:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575475.909843:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575475.909845:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488902016:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575475.909850:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488902144:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575475.909870:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575475.909882:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575475/real 1788575475] req@ffff94d408c57100 x1875450488902144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575475.909904:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488902144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575475.909914:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575475.909918:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575475.909922:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488902144:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575475.909936:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488902272:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575475.909961:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575475.909973:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575475/real 1788575475] req@ffff94d408c54a80 x1875450488902272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575475.909987:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488902272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575475.909995:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575475.909999:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575475.910003:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488902272:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575475.910009:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488902400:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575475.910026:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575475.910034:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575475/real 1788575475] req@ffff94d408c56d80 x1875450488902400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575475.910041:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488902400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575475.910052:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575475.910054:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575475.910056:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488902400:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575476.421357:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575476.869416:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575477.445367:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575477.893430:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575478.469331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575478.917374:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575479.493275:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575479.941464:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575480.517321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575480.965478:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575481.029335:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575481.029346:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575481.029352:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575481.029361:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6629 00000100:00080000:3.0:1788575481.029373:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575481.029381:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575481.029419:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488902528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575481.029453:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575481.029458:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575481.029460:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575481.029464:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6629 00000100:00080000:3.0:1788575481.029468:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575481.029472:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575481.029483:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575481.029499:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488902656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575481.029503:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488902528:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575481.029519:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575481.029523:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575481.029526:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575481.029532:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6629 00000100:00080000:3.0:1788575481.029536:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575481.029541:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575481.029547:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575481.029564:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488902784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575481.029575:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575481.029579:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575481.029582:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575481.029586:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6629 00000100:00100000:0.0:1788575481.029591:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488902656:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788575481.029592:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575481.029602:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575481.029609:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575481.029627:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488902784:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575481.029627:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488902912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575481.029644:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575481.029648:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575481.029651:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575481.029673:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6629 00000100:00100000:0.0:1788575481.029677:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488902912:192.168.202.125@tcp:8:kworker.0 00000100:00080000:3.0:1788575481.029680:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575481.029689:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575481.029695:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575481.029713:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488903040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575481.029739:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488903040:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575481.541293:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575481.989358:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575482.565330:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575483.013388:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575483.589276:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575484.037323:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788575484.101476:0:2392:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:2.0:1788575484.101516:0:2392:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788575484.101531:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788575484.101572:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788575484.101580:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788575484.101588:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788575484.101595:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:0.0:1788575484.101678:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575481/real 1788575484] req@ffff94d408c57100 x1875450488903040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575484.101704:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488903040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575484.101716:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575484.101722:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575484.101726:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488903040:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575484.101734:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575481/real 1788575484] req@ffff94d408c54a80 x1875450488902912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575484.101753:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488902912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575484.101761:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575484.101765:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575484.101768:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488902912:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575484.101775:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575481/real 1788575484] req@ffff94d408c56680 x1875450488902656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575484.101786:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488902656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575484.101793:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575484.101796:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575484.101799:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488902656:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575484.101806:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575481/real 1788575484] req@ffff94d408c56d80 x1875450488902784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575484.101816:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488902784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575484.101824:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575484.101827:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575484.101830:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488902784:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575484.101836:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575481/real 1788575484] req@ffff94d408c57b80 x1875450488902528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575484.101848:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488902528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575484.101855:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575484.101858:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575484.101861:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488902528:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575484.613273:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575485.061342:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575485.637337:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575486.085347:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575486.149497:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575486.149511:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575486.149516:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575486.149529:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6634 00000100:00080000:3.0:1788575486.149547:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575486.149555:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575486.149600:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488903168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575486.149648:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575486.149652:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575486.149654:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575486.149658:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6634 00000100:00080000:3.0:1788575486.149662:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575486.149666:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575486.149679:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575486.149693:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488903296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575486.149703:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575486.149707:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575486.149709:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575486.149714:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6634 00000100:00080000:3.0:1788575486.149718:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575486.149721:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575486.149728:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575486.149743:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488903424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575486.149751:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488903168:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575486.149757:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575486.149767:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575486.149769:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575486.149774:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6634 00000100:00080000:3.0:1788575486.149777:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575486.149781:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575486.149788:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575486.149806:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488903552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575486.149817:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575486.149821:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575486.149823:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575486.149827:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6634 00000100:00080000:3.0:1788575486.149831:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575486.149836:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575486.149841:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788575486.149856:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575486.149856:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488903680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575486.149921:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575486/real 1788575486] req@ffff94d408c55c00 x1875450488903168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575486.149946:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488903168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575486.149960:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575486.149964:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575486.149968:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488903168:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575486.149978:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488903296:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575486.150003:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575486.150016:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575486/real 1788575486] req@ffff94d408c54e00 x1875450488903296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575486.150027:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488903296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575486.150034:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575486.150038:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575486.150040:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488903296:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575486.150069:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488903424:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575486.150109:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575486.150120:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575486/real 1788575486] req@ffff94d408c56a00 x1875450488903424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575486.150130:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488903424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575486.150143:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575486.150158:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575486.150162:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488903424:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575486.150170:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488903552:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575486.150192:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575486.150280:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575486/real 1788575486] req@ffff94d408c57800 x1875450488903552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575486.150293:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488903552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575486.150300:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575486.150303:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575486.150307:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488903552:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575486.150314:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488903680:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575486.150335:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575486.150347:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575486/real 1788575486] req@ffff94d408c54380 x1875450488903680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575486.150358:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488903680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575486.150365:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575486.150368:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575486.150376:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488903680:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575486.661355:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575487.109415:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575487.685266:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575488.133390:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575488.709324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575489.157405:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575489.733389:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575490.181469:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575490.757390:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575491.205478:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575491.269314:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575491.269324:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575491.269328:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575491.269337:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6639 00000100:00080000:3.0:1788575491.269347:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575491.269353:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575491.269388:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488903808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575491.269423:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575491.269426:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575491.269428:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575491.269431:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6639 00000100:00080000:3.0:1788575491.269434:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575491.269438:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575491.269447:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575491.269460:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488903808:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575491.269463:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488903936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575491.269474:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575491.269477:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575491.269478:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575491.269481:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6639 00000100:00080000:3.0:1788575491.269485:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575491.269488:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575491.269493:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575491.269504:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488904064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575491.269511:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788575491.269512:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575491.269516:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575491.269518:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575491.269521:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6639 00000100:00080000:3.0:1788575491.269524:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575491.269527:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575491.269531:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575491.269539:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488904192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575491.269545:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575491/real 1788575491] req@ffff94d408c55880 x1875450488903808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575491.269553:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575491.269558:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575491.269560:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575491.269562:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488903808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575491.269567:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6639 00000100:00080000:0.0:1788575491.269573:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575491.269574:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575491.269578:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:3.0:1788575491.269582:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575491.269583:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488903808:192.168.202.125@tcp:250:kworker.0 00000008:00100000:3.0:1788575491.269592:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575491.269602:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488904320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575491.269606:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488903936:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575491.269627:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575491.269637:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575491/real 1788575491] req@ffff94d408c55180 x1875450488903936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575491.269645:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488903936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575491.269651:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575491.269654:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575491.269656:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488903936:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575491.269660:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488904064:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575491.269668:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575491.269673:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575491/real 1788575491] req@ffff94d408c54700 x1875450488904064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575491.269680:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488904064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575491.269684:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575491.269686:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575491.269688:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488904064:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575491.269692:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488904192:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575491.269707:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575491.269716:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575491/real 1788575491] req@ffff94d408c56300 x1875450488904192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575491.269728:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488904192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575491.269742:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575491.269744:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575491.269746:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488904192:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575491.269757:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488904320:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575491.269770:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575491.269781:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575491/real 1788575491] req@ffff94d408c55f80 x1875450488904320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575491.269788:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488904320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575491.269793:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575491.269794:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575491.269796:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488904320:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575491.781407:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575492.229409:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575492.806248:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575493.253393:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575493.830849:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575494.277356:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575494.853305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575495.301327:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575495.877447:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575496.325381:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575496.389388:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575496.389399:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575496.389403:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575496.389413:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6644 00000100:00080000:1.0:1788575496.389424:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575496.389432:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575496.389473:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488904448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575496.389507:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575496.389510:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575496.389511:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575496.389514:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6644 00000100:00080000:1.0:1788575496.389516:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575496.389518:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575496.389526:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575496.389535:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd880 x1875450488904576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575496.389541:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575496.389544:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575496.389546:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575496.389548:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6644 00000100:00080000:1.0:1788575496.389550:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575496.389552:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575496.389555:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575496.389565:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488904448:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575496.389565:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488904704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575496.389579:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575496.389581:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575496.389582:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575496.389584:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6644 00000100:00080000:1.0:1788575496.389586:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575496.389589:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575496.389592:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575496.389599:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488904832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575496.389604:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575496.389607:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575496.389608:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575496.389610:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6644 00000100:00080000:1.0:1788575496.389613:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575496.389615:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575496.389617:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575496.389623:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488904960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575496.389659:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575496.389702:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575496/real 1788575496] req@ffff94d4049bca80 x1875450488904448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575496.389727:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488904448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575496.389736:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575496.389739:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575496.389743:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488904448:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575496.389748:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488904576:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575496.389768:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575496.389780:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575496/real 1788575496] req@ffff94d4049bd880 x1875450488904576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575496.389792:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd880 x1875450488904576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575496.389800:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575496.389803:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575496.389807:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488904576:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575496.389833:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488904704:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575496.389854:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575496.389865:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575496/real 1788575496] req@ffff94d4049bd500 x1875450488904704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575496.389872:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488904704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575496.389878:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575496.389879:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575496.389882:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488904704:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575496.389887:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488904832:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575496.389906:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575496.389917:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575496/real 1788575496] req@ffff94d4049bdc00 x1875450488904832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575496.389928:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488904832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575496.389936:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575496.389939:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575496.389941:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488904832:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575496.389945:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488904960:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575496.389958:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575496.389965:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575496/real 1788575496] req@ffff94d4049be680 x1875450488904960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575496.389972:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488904960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575496.389977:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575496.389979:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575496.389980:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488904960:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575496.901349:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575497.349393:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575497.925364:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575498.373403:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575498.949365:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575499.397352:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575499.973348:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575500.421419:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575500.997342:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575501.445319:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575501.509370:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575501.509381:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575501.509385:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575501.509396:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6649 00000100:00080000:3.0:1788575501.509414:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575501.509424:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575501.509470:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488905088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575501.509497:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575501.509500:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575501.509501:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575501.509504:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6649 00000100:00080000:3.0:1788575501.509507:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575501.509510:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575501.509530:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575501.509542:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488905216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575501.509548:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488905088:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575501.509551:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575501.509557:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575501.509559:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575501.509564:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6649 00000100:00080000:3.0:1788575501.509567:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575501.509571:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575501.509576:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575501.509590:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488905344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575501.509601:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575501.509604:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575501.509607:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575501.509611:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6649 00000100:00080000:3.0:1788575501.509615:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575501.509618:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575501.509623:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575501.509632:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488905472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575501.509637:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575501.509653:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575501.509658:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575501.509660:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575501.509679:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575501/real 1788575501] req@ffff94d408c57480 x1875450488905088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575501.509705:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488905088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575501.509717:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575501.509723:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:3.0:1788575501.509724:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6649 00000100:00080000:3.0:1788575501.509729:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575501.509730:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488905088:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575501.509736:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575501.509742:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575501.509762:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488905216:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575501.509786:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575501.509805:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575501/real 1788575501] req@ffff94d408c54000 x1875450488905216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575501.509817:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488905216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575501.509826:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575501.509830:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575501.509834:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488905216:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575501.509840:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488905344:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575501.509863:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575501.509870:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488905600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575501.509876:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575501/real 1788575501] req@ffff94d408c55f80 x1875450488905344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575501.509893:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488905344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575501.509899:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575501.509901:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575501.509904:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488905344:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575501.509908:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488905472:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575501.509926:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575501.509933:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575501/real 1788575501] req@ffff94d408c56300 x1875450488905472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575501.509941:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488905472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575501.509947:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575501.509950:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575501.509954:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488905472:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575501.509973:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488905600:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575501.509992:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575501.510003:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575501/real 1788575501] req@ffff94d408c54700 x1875450488905600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575501.510014:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488905600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575501.510021:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575501.510023:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575501.510025:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488905600:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575502.021324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575502.469389:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575503.045276:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575503.493549:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575504.069316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575504.517343:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575505.093288:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575505.541357:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575506.117300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575506.565316:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575506.629297:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575506.629305:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575506.629308:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575506.629316:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6654 00000100:00080000:1.0:1788575506.629324:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575506.629329:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575506.629372:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488905728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575506.629398:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575506.629400:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575506.629401:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575506.629404:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6654 00000100:00080000:1.0:1788575506.629407:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575506.629409:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575506.629421:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575506.629430:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488905856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575506.629439:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488905728:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788575506.629440:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575506.629448:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575506.629450:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575506.629454:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6654 00000100:00080000:1.0:1788575506.629458:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575506.629462:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575506.629467:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575506.629479:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488905984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575506.629488:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575506.629490:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1788575506.629496:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788575506.629498:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575506.629503:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6654 00000100:00080000:1.0:1788575506.629507:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575506.629510:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575506.629514:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575506.629523:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488906112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575506.629529:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575506.629531:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575506.629532:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575506.629536:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6654 00000100:00100000:0.0:1788575506.629538:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575506/real 1788575506] req@ffff94d4049bf100 x1875450488905728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575506.629538:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575506.629543:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575506.629548:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575506.629556:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488906240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575506.629568:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488905728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575506.629580:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575506.629595:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575506.629602:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488905728:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575506.629625:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488905856:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575506.629649:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575506.629676:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575506/real 1788575506] req@ffff94d4049be300 x1875450488905856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575506.629689:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488905856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575506.629702:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575506.629705:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575506.629708:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488905856:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575506.629714:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488905984:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575506.629732:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575506.629742:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575506/real 1788575506] req@ffff94d4049bf800 x1875450488905984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575506.629752:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488905984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575506.629763:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575506.629766:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575506.629769:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488905984:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575506.629774:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488906112:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575506.629791:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575506.629800:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575506/real 1788575506] req@ffff94d4049bed80 x1875450488906112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575506.629809:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488906112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575506.629816:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575506.629818:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575506.629821:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488906112:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575506.629826:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488906240:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575506.629840:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575506.629850:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575506/real 1788575506] req@ffff94d4049bc700 x1875450488906240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575506.629859:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488906240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575506.629866:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575506.629868:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575506.629872:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488906240:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575507.141275:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575507.589403:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575508.165298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575508.613361:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575509.189309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575509.637398:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575510.213277:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575510.661360:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575511.237327:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575511.685340:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575511.749376:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575511.749391:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575511.749397:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575511.749409:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6660 00000100:00080000:3.0:1788575511.749424:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575511.749434:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575511.749493:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488906368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575511.749527:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575511.749538:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575511.749540:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575511.749545:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6660 00000100:00080000:3.0:1788575511.749549:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575511.749554:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575511.749565:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575511.749576:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488906368:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575511.749582:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488906496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575511.749596:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575511.749600:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575511.749602:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575511.749606:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6660 00000100:00080000:3.0:1788575511.749610:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575511.749615:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575511.749620:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575511.749633:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488906624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575511.749648:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575511.749656:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575511.749660:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575511.749663:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575511.749668:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6660 00000100:00080000:3.0:1788575511.749675:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575511.749679:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575511.749686:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575511.749691:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575511/real 1788575511] req@ffff94d408c55180 x1875450488906368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575511.749701:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488906752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575511.749708:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488906368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575511.749718:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575511.749722:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575511.749723:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575511.749728:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:3.0:1788575511.749728:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575511.749737:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488906368:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575511.749738:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6660 00000100:00080000:3.0:1788575511.749746:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575511.749752:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575511.749758:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575511.749765:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488906496:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575511.749772:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488906880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575511.749784:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575511.749795:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575511/real 1788575511] req@ffff94d408c55880 x1875450488906496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575511.749803:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488906496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575511.749823:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575511.749826:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575511.749828:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488906496:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575511.749833:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488906624:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575511.749847:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575511.749854:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575511/real 1788575511] req@ffff94d408c54380 x1875450488906624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575511.749862:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488906624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575511.749867:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575511.749869:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575511.749871:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488906624:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575511.749874:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488906752:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575511.749884:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575511.749889:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575511/real 1788575511] req@ffff94d408c57800 x1875450488906752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575511.749895:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488906752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575511.749900:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575511.749901:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575511.749903:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488906752:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575511.749912:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488906880:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575511.749921:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575511.749926:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575511/real 1788575511] req@ffff94d408c56a00 x1875450488906880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575511.749937:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488906880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575511.749942:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575511.749943:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575511.749945:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488906880:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575512.261322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575512.709377:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575513.285320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575513.733345:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575514.309299:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575514.757345:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575515.333308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575515.781353:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575516.357305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575516.805354:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575516.869312:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575516.869321:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575516.869324:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575516.869332:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6665 00000100:00080000:1.0:1788575516.869340:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575516.869346:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575516.869373:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488907008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575516.869392:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575516.869395:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575516.869396:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575516.869398:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6665 00000100:00080000:1.0:1788575516.869400:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575516.869402:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575516.869409:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575516.869417:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488907136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575516.869424:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575516.869426:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575516.869427:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575516.869432:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488907008:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788575516.869432:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6665 00000100:00080000:1.0:1788575516.869437:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575516.869441:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575516.869443:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575516.869449:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488907264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575516.869459:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575516.869461:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575516.869462:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575516.869465:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6665 00000100:00080000:1.0:1788575516.869475:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575516.869478:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575516.869481:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788575516.869487:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575516.869489:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488907392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575516.869496:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575516.869498:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575516.869500:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575516.869502:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6665 00000100:00080000:1.0:1788575516.869506:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575516.869510:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575516.869514:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575516.869527:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575516/real 1788575516] req@ffff94d4049bc000 x1875450488907008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575516.869528:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488907520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575516.869569:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488907008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575516.869588:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575516.869596:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575516.869600:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488907008:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575516.869619:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488907136:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575516.869635:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575516.869644:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575516/real 1788575516] req@ffff94d4049bfb80 x1875450488907136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575516.869651:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488907136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575516.869660:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575516.869662:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575516.869664:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488907136:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575516.869668:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488907264:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575516.869677:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575516.869682:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575516/real 1788575516] req@ffff94d4049bd180 x1875450488907264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575516.869689:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488907264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575516.869701:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575516.869703:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575516.869704:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488907264:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575516.869708:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488907392:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575516.869718:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575516.869724:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575516/real 1788575516] req@ffff94d4049bce00 x1875450488907392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575516.869731:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488907392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575516.869735:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575516.869737:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575516.869739:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488907392:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575516.869742:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488907520:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575516.869750:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575516.869755:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575516/real 1788575516] req@ffff94d4049bea00 x1875450488907520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575516.869761:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488907520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575516.869766:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575516.869767:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575516.869769:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488907520:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575517.381309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575517.829382:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575518.405290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575518.853333:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575519.429327:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575519.877358:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575520.453298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575520.901334:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575521.477299:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575521.925362:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575521.989323:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575521.989330:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575521.989333:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575521.989341:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6670 00000100:00080000:3.0:1788575521.989349:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575521.989354:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575521.989389:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488907648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575521.989422:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575521.989426:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575521.989428:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575521.989432:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6670 00000100:00080000:3.0:1788575521.989436:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575521.989441:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575521.989449:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575521.989457:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488907648:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575521.989465:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488907776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575521.989476:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575521.989479:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575521.989482:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575521.989486:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6670 00000100:00080000:3.0:1788575521.989490:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575521.989493:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575521.989498:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575521.989508:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488907904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575521.989510:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575521.989519:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575521.989521:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575521.989523:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575521.989527:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6670 00000100:00080000:3.0:1788575521.989530:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575521.989533:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575521.989537:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575521.989544:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575521/real 1788575521] req@ffff94d408c54e00 x1875450488907648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575521.989547:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488908032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575521.989568:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575521.989569:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575521.989571:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575521.989574:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488907648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575521.989575:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6670 00000100:00080000:3.0:1788575521.989579:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575521.989585:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00080000:0.0:1788575521.989590:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788575521.989590:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788575521.989596:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575521.989599:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488908160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575521.989602:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488907648:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575521.989628:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488907776:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575521.989645:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575521.989652:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575521/real 1788575521] req@ffff94d408c55c00 x1875450488907776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575521.989660:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488907776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575521.989664:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575521.989666:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575521.989668:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488907776:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575521.989671:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488907904:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575521.989680:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575521.989685:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575521/real 1788575521] req@ffff94d408c57b80 x1875450488907904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575521.989691:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488907904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575521.989696:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575521.989697:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575521.989699:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488907904:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575521.989702:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488908032:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575521.989710:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575521.989715:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575521/real 1788575521] req@ffff94d408c56d80 x1875450488908032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575521.989730:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488908032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575521.989737:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575521.989739:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575521.989742:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488908032:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575521.989747:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488908160:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575521.989765:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575521.989774:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575521/real 1788575521] req@ffff94d408c56680 x1875450488908160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575521.989781:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488908160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575521.989786:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575521.989788:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575521.989790:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488908160:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575522.501274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575522.949350:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575523.525299:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575523.973376:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575524.549299:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575524.997365:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575525.573271:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575526.021332:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575526.597307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575527.045371:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575527.109289:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575527.109300:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575527.109304:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575527.109314:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6675 00000100:00080000:3.0:1788575527.109324:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575527.109330:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575527.109360:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488908288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575527.109384:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575527.109387:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575527.109389:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575527.109393:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6675 00000100:00080000:3.0:1788575527.109396:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575527.109401:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575527.109408:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575527.109421:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488908416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575527.109423:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488908288:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575527.109449:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575527.109453:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575527.109455:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575527.109459:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6675 00000100:00080000:3.0:1788575527.109463:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575527.109468:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575527.109473:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788575527.109478:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575527.109489:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488908544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575527.109499:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575527.109503:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575527.109505:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575527.109508:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575527/real 1788575527] req@ffff94d408c54a80 x1875450488908288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575527.109512:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6675 00000100:00080000:3.0:1788575527.109519:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575527.109527:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575527.109533:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575527.109537:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488908288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575527.109547:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488908672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575527.109553:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575527.109560:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575527.109561:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:3.0:1788575527.109567:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1788575527.109568:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488908288:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575527.109572:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575527.109578:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6675 00000100:00080000:3.0:1788575527.109582:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575527.109585:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575527.109591:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575527.109593:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488908416:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575527.109611:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488908800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575527.109621:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575527.109637:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575527/real 1788575527] req@ffff94d408c57100 x1875450488908416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575527.109646:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488908416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575527.109651:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575527.109653:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575527.109655:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488908416:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575527.109658:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488908544:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575527.109671:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575527.109677:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575527/real 1788575527] req@ffff94d408c56680 x1875450488908544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575527.109683:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488908544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575527.109688:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575527.109690:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575527.109692:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488908544:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575527.109695:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488908672:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575527.109706:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575527.109714:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575527/real 1788575527] req@ffff94d408c56d80 x1875450488908672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575527.109724:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488908672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575527.109731:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575527.109733:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575527.109736:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488908672:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575527.109751:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488908800:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575527.109776:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575527.109785:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575527/real 1788575527] req@ffff94d408c57b80 x1875450488908800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575527.109795:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488908800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575527.109800:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575527.109801:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575527.109803:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488908800:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575527.621284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575528.069376:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575528.645312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575529.093376:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575529.669316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575530.117445:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575530.693285:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575531.141448:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575531.717292:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575532.165431:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575532.229358:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575532.229369:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575532.229373:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575532.229384:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6680 00000100:00080000:1.0:1788575532.229397:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575532.229404:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575532.229457:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488908928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575532.229488:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575532.229491:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575532.229493:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575532.229496:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6680 00000100:00080000:1.0:1788575532.229499:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575532.229502:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575532.229512:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575532.229522:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488909056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575532.229529:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575532.229533:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575532.229536:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575532.229538:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488908928:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788575532.229540:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6680 00000100:00080000:1.0:1788575532.229543:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575532.229548:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575532.229550:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575532.229567:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488909184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575532.229573:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575532.229576:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575532.229577:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575532.229580:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6680 00000100:00080000:1.0:1788575532.229583:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575532.229585:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575532.229589:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575532.229597:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488909312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575532.229603:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575532.229606:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575532.229607:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575532.229608:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6680 00000100:00080000:1.0:1788575532.229610:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575532.229612:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575532.229615:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788575532.229616:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575532.229624:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488909440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575532.229664:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575532/real 1788575532] req@ffff94d4049bf480 x1875450488908928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575532.229679:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488908928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575532.229687:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575532.229690:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575532.229693:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488908928:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575532.229714:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488909056:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575532.229728:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575532.229737:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575532/real 1788575532] req@ffff94d4049bc380 x1875450488909056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575532.229744:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc380 x1875450488909056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575532.229749:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575532.229751:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575532.229753:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488909056:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575532.229756:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488909184:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575532.229766:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575532.229771:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575532/real 1788575532] req@ffff94d4049bea00 x1875450488909184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575532.229778:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488909184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575532.229786:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575532.229789:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575532.229791:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488909184:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575532.229793:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488909312:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575532.229803:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575532.229809:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575532/real 1788575532] req@ffff94d4049bce00 x1875450488909312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575532.229819:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488909312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575532.229826:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575532.229829:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575532.229832:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488909312:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575532.229838:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488909440:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575532.229855:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575532.229863:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575532/real 1788575532] req@ffff94d4049bd180 x1875450488909440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575532.229870:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488909440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575532.229875:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575532.229877:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575532.229879:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488909440:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575532.741343:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575533.189416:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575533.765366:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575534.213398:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575534.789364:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575535.237406:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575535.813356:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575536.261390:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575536.837373:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575537.285410:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575537.349413:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575537.349425:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575537.349432:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575537.349445:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6685 00000100:00080000:3.0:1788575537.349456:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575537.349464:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575537.349501:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488909568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575537.349532:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575537.349537:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575537.349539:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575537.349544:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6685 00000100:00080000:3.0:1788575537.349549:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575537.349554:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575537.349566:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575537.349571:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488909568:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575537.349581:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488909696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575537.349591:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575537.349593:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575537.349595:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575537.349598:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6685 00000100:00080000:3.0:1788575537.349602:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575537.349605:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575537.349610:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575537.349618:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488909824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575537.349629:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575537.349632:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575537.349635:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788575537.349640:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575537.349642:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6685 00000100:00080000:3.0:1788575537.349647:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575537.349680:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575537/real 1788575537] req@ffff94d408c55c00 x1875450488909568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575537.349697:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488909568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575537.349706:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575537.349709:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:3.0:1788575537.349713:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575537.349714:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488909568:192.168.202.125@tcp:250:kworker.0 00000008:00100000:3.0:1788575537.349721:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575537.349733:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488909952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575537.349742:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488909696:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575537.349760:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575537.349768:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575537/real 1788575537] req@ffff94d408c54e00 x1875450488909696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575537.349776:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488909696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575537.349781:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575537.349783:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575537.349785:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488909696:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575537.349789:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488909824:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575537.349801:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575537.349807:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575537/real 1788575537] req@ffff94d408c56a00 x1875450488909824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575537.349815:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488909824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575537.349824:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575537.349827:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575537.349832:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488909824:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788575537.349837:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575537.349841:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575537.349842:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575537.349845:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488909952:192.168.202.125@tcp:8:kworker.0 00000100:00080000:3.0:1788575537.349849:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6685 00000800:00000100:1.0:1788575537.349863:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575537.349870:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575537/real 1788575537] req@ffff94d408c57800 x1875450488909952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575537.349881:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575537.349886:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575537.349890:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488909952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788575537.349890:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788575537.349900:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788575537.349903:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488910080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575537.349906:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575537.349913:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488909952:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575537.349929:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488910080:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575537.349946:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575537.349954:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575537/real 1788575537] req@ffff94d408c54380 x1875450488910080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575537.349965:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488910080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575537.349971:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575537.349974:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575537.349976:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488910080:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575537.861299:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575538.309346:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575538.885295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575539.333365:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575539.909302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575540.357379:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575540.933308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575541.381389:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575541.957298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575542.405342:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575542.469363:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575542.469378:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575542.469384:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575542.469398:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6690 00000100:00080000:2.0:1788575542.469410:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575542.469418:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575542.469485:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488910208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575542.469510:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575542.469513:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575542.469514:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575542.469518:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6690 00000100:00080000:2.0:1788575542.469520:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575542.469523:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575542.469532:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575542.469542:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488910336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575542.469554:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575542.469556:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575542.469558:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575542.469562:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6690 00000100:00100000:1.0:1788575542.469564:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488910208:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788575542.469565:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575542.469571:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575542.469575:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575542.469583:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488910464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575542.469588:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575542.469590:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575542.469591:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575542.469594:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6690 00000100:00080000:2.0:1788575542.469596:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575542.469598:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575542.469601:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575542.469608:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488910592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575542.469614:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575542.469616:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575542.469617:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575542.469619:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6690 00000100:00080000:2.0:1788575542.469622:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575542.469625:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575542.469628:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575542.469646:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488910720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575542.469652:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575542.469708:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575542/real 1788575542] req@ffff94d408c61180 x1875450488910208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575542.469728:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488910208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575542.469740:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575542.469744:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575542.469748:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488910208:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575542.469785:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488910336:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575542.469804:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575542.469813:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575542/real 1788575542] req@ffff94d408c62d80 x1875450488910336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575542.469821:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488910336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575542.469827:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575542.469829:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575542.469831:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488910336:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575542.469834:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488910464:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575542.469845:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575542.469851:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575542/real 1788575542] req@ffff94d408c60000 x1875450488910464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575542.469859:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488910464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575542.469864:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575542.469865:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575542.469867:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488910464:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575542.469870:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488910592:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575542.469881:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575542.469886:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575542/real 1788575542] req@ffff94d408c63800 x1875450488910592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575542.469893:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488910592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575542.469897:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575542.469899:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575542.469900:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488910592:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575542.469903:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488910720:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575542.469912:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575542.469917:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575542/real 1788575542] req@ffff94d408c62680 x1875450488910720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575542.469928:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488910720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575542.469933:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575542.469934:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575542.469936:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488910720:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575542.981363:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575543.429425:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575544.005263:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575544.453404:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575545.029351:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575545.477338:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575546.053300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575546.501466:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575547.077318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575547.525429:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575547.589372:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575547.589382:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575547.589387:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575547.589396:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6695 00000100:00080000:3.0:1788575547.589406:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575547.589414:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575547.589450:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488910848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575547.589483:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575547.589486:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575547.589488:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575547.589492:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6695 00000100:00080000:3.0:1788575547.589496:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575547.589500:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575547.589508:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575547.589537:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488910976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575547.589547:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575547.589550:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575547.589552:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575547.589556:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6695 00000100:00080000:3.0:1788575547.589560:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575547.589564:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488910848:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575547.589568:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575547.589575:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575547.589590:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488911104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575547.589606:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575547.589609:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575547.589612:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575547.589616:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6695 00000100:00080000:3.0:1788575547.589620:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575547.589624:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575547.589630:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575547.589642:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488911232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575547.589652:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575547.589655:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488910976:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788575547.589657:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575547.589662:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575547.589668:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6695 00000100:00080000:3.0:1788575547.589672:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575547.589676:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575547.589681:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575547.589697:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488911360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575547.589701:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488911104:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575547.589737:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488911232:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575547.589761:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488911360:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575548.101369:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575548.549441:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575549.125338:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575549.573433:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575550.149349:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575550.597421:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788575550.661502:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:2.0:1788575550.661553:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788575550.661587:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788575550.661630:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788575550.661639:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788575550.661648:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788575550.661656:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:1.0:1788575550.661785:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575547/real 1788575550] req@ffff94d408c55f80 x1875450488911360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575550.661826:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488911360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575550.661840:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575550.661846:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575550.661852:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488911360:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575550.661863:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575547/real 1788575550] req@ffff94d408c54700 x1875450488911104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575550.661878:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488911104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575550.661902:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575550.661906:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575550.661910:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488911104:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575550.661918:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575547/real 1788575550] req@ffff94d408c56300 x1875450488911232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575550.661938:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488911232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575550.661946:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575550.661949:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575550.661952:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488911232:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575550.661960:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575547/real 1788575550] req@ffff94d408c55880 x1875450488910848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575550.661974:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488910848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575550.661983:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575550.661986:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575550.661989:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488910848:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575550.662000:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575547/real 1788575550] req@ffff94d408c55180 x1875450488910976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575550.662012:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488910976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575550.662019:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575550.662023:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575550.662026:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488910976:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575551.173375:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575551.621426:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575552.197361:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575552.645411:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575552.709284:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575552.709294:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575552.709299:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575552.709310:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6701 00000100:00080000:3.0:1788575552.709321:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575552.709329:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575552.709370:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488911488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575552.709409:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575552.709413:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575552.709415:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575552.709419:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6701 00000100:00080000:3.0:1788575552.709423:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575552.709427:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575552.709439:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575552.709453:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488911616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575552.709464:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488911488:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575552.709466:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575552.709474:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575552.709477:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575552.709482:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6701 00000100:00080000:3.0:1788575552.709485:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575552.709488:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575552.709492:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575552.709503:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488911744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575552.709514:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575552.709516:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575552.709518:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575552.709522:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6701 00000100:00080000:3.0:1788575552.709526:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575552.709530:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575552.709535:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575552.709546:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488911872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575552.709555:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788575552.709559:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575552.709561:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575552.709564:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575552.709568:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6701 00000100:00080000:3.0:1788575552.709574:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575552.709578:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575552.709584:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575552.709594:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488912000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575552.709644:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575552/real 1788575552] req@ffff94d408c54000 x1875450488911488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575552.709668:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488911488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575552.709684:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575552.709688:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575552.709692:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488911488:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575552.709723:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488911616:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575552.709750:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575552.709764:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575552/real 1788575552] req@ffff94d408c57480 x1875450488911616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575552.709776:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488911616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575552.709785:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575552.709789:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575552.709792:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488911616:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575552.709798:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488911744:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575552.709820:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575552.709832:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575552/real 1788575552] req@ffff94d408c55180 x1875450488911744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575552.709842:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488911744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575552.709851:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575552.709854:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575552.709857:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488911744:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575552.709862:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488911872:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575552.709881:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575552.709891:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575552/real 1788575552] req@ffff94d408c55880 x1875450488911872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575552.709902:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488911872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575552.709909:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575552.709912:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575552.709915:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488911872:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575552.709921:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488912000:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575552.709942:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575552.709954:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575552/real 1788575552] req@ffff94d408c56300 x1875450488912000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575552.709963:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488912000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575552.709971:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575552.709974:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575552.709977:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488912000:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575553.221356:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575553.669383:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575554.245339:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575554.693330:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575555.269291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575555.717397:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575556.293306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575556.741342:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575557.317385:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575557.765388:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575557.829287:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575557.829295:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575557.829298:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575557.829305:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6706 00000100:00080000:1.0:1788575557.829313:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575557.829318:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575557.829350:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488912128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575557.829381:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575557.829384:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575557.829385:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575557.829389:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6706 00000100:00080000:1.0:1788575557.829391:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575557.829394:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575557.829401:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575557.829411:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488912256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575557.829418:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575557.829422:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488912128:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788575557.829422:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575557.829425:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575557.829431:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6706 00000100:00080000:1.0:1788575557.829433:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575557.829435:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575557.829437:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575557.829443:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488912384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575557.829449:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575557.829450:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575557.829451:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575557.829454:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6706 00000100:00080000:1.0:1788575557.829456:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575557.829458:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575557.829460:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575557.829466:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488912512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575557.829472:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575557.829474:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575557.829475:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575557.829477:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6706 00000100:00080000:1.0:1788575557.829480:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788575557.829483:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788575557.829484:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575557.829488:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575557.829496:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488912640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575557.829515:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575557/real 1788575557] req@ffff94d4049bfb80 x1875450488912128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575557.829527:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488912128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575557.829534:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575557.829536:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575557.829539:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488912128:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575557.829554:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488912256:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575557.829567:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575557.829574:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575557/real 1788575557] req@ffff94d4049bc000 x1875450488912256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575557.829580:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488912256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575557.829585:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575557.829587:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575557.829589:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488912256:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575557.829592:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488912384:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575557.829601:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575557.829605:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575557/real 1788575557] req@ffff94d4049bc700 x1875450488912384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575557.829612:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488912384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575557.829616:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575557.829618:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575557.829619:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488912384:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575557.829622:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488912512:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575557.829630:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575557.829635:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575557/real 1788575557] req@ffff94d4049bed80 x1875450488912512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575557.829640:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488912512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575557.829644:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575557.829646:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575557.829648:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488912512:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575557.829651:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488912640:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575557.829659:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575557.829663:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575557/real 1788575557] req@ffff94d4049bf800 x1875450488912640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575557.829669:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488912640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575557.829673:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575557.829674:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575557.829676:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488912640:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575558.343143:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575558.789353:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575559.365843:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575559.813374:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575560.389362:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575560.837362:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575561.414517:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575561.861392:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575562.437358:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575562.885460:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575562.949338:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575562.949348:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575562.949353:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575562.949360:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6711 00000100:00080000:3.0:1788575562.949368:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575562.949373:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575562.949408:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488912768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575562.949434:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575562.949438:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575562.949440:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575562.949444:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6711 00000100:00080000:3.0:1788575562.949447:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575562.949451:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575562.949459:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575562.949472:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488912896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575562.949482:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575562.949486:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488912768:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575562.949487:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575562.949490:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575562.949498:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6711 00000100:00080000:3.0:1788575562.949502:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575562.949506:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575562.949512:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575562.949524:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488913024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575562.949551:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788575562.949552:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575562.949557:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575562.949560:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575562.949565:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6711 00000100:00080000:3.0:1788575562.949570:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575562.949574:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575562.949580:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575562.949592:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488913152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575562.949596:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575562/real 1788575562] req@ffff94d408c54700 x1875450488912768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575562.949608:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575562.949614:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575562.949617:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575562.949621:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6711 00000100:00080000:3.0:1788575562.949625:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575562.949628:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575562.949631:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575562.949632:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488912768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575562.949644:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488913280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575562.949646:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575562.949655:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575562.949662:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488912768:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575562.949685:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488912896:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575562.949710:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575562.949723:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575562/real 1788575562] req@ffff94d408c55f80 x1875450488912896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575562.949735:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488912896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575562.949743:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575562.949746:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575562.949749:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488912896:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575562.949755:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488913024:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575562.949787:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575562.949799:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575562/real 1788575562] req@ffff94d408c54380 x1875450488913024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575562.949810:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488913024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575562.949818:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575562.949821:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575562.949824:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488913024:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575562.949830:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488913152:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575562.949846:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575562.949854:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575562/real 1788575562] req@ffff94d408c57800 x1875450488913152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575562.949872:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488913152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575562.949879:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575562.949882:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575562.949884:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488913152:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575562.949889:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488913280:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575562.949902:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575562.949910:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575562/real 1788575562] req@ffff94d408c56a00 x1875450488913280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575562.949920:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488913280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575562.949927:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575562.949929:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575562.949932:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488913280:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575563.461303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575563.909328:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575564.485303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575564.933349:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575565.509295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575565.957359:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575566.533300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575566.981328:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575567.557346:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575568.005371:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575568.069320:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575568.069330:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575568.069333:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575568.069341:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6716 00000100:00080000:1.0:1788575568.069349:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575568.069354:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575568.069386:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488913408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575568.069413:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575568.069416:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575568.069418:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575568.069424:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6716 00000100:00080000:1.0:1788575568.069431:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575568.069436:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575568.069444:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575568.069456:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488913536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575568.069466:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575568.069471:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488913408:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788575568.069471:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575568.069476:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575568.069481:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6716 00000100:00080000:1.0:1788575568.069485:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575568.069489:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575568.069494:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575568.069505:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488913664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575568.069511:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575568.069513:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575568.069514:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575568.069517:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6716 00000100:00080000:1.0:1788575568.069520:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575568.069523:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575568.069526:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788575568.069533:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575568.069537:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488913792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575568.069547:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575568.069550:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575568.069553:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575568.069557:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6716 00000100:00080000:1.0:1788575568.069561:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575568.069565:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575568.069570:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575568.069574:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575568/real 1788575568] req@ffff94d4049be300 x1875450488913408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575568.069581:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488913920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575568.069608:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488913408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575568.069620:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575568.069623:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575568.069627:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488913408:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575568.069645:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488913536:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575568.069661:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575568.069669:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575568/real 1788575568] req@ffff94d4049bf100 x1875450488913536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575568.069676:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488913536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575568.069681:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575568.069683:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575568.069686:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488913536:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575568.069692:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488913664:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575568.069709:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575568.069720:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575568/real 1788575568] req@ffff94d4049be680 x1875450488913664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575568.069732:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488913664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575568.069739:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575568.069742:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575568.069746:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488913664:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575568.069752:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488913792:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575568.069774:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575568.069784:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575568/real 1788575568] req@ffff94d4049bdc00 x1875450488913792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575568.069794:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488913792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575568.069802:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575568.069805:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575568.069808:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488913792:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575568.069813:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488913920:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575568.069826:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575568.069833:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575568/real 1788575568] req@ffff94d4049bd500 x1875450488913920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575568.069870:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488913920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575568.069878:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575568.069881:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575568.069884:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488913920:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575568.581268:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575569.029328:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575569.605345:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575570.053377:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575570.629282:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575571.077407:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575571.653320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575572.101391:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575572.677308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575573.125410:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575573.189359:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575573.189374:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575573.189379:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575573.189391:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6721 00000100:00080000:1.0:1788575573.189404:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575573.189414:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575573.189460:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd880 x1875450488914048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575573.189495:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575573.189499:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575573.189501:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575573.189506:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6721 00000100:00080000:1.0:1788575573.189511:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575573.189516:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575573.189526:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575573.189540:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488914048:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575573.189543:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488914176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575573.189559:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575573.189563:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575573.189565:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575573.189569:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6721 00000100:00080000:1.0:1788575573.189572:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575573.189576:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575573.189581:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575573.189596:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488914304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575573.189606:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575573.189609:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575573.189611:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575573.189615:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6721 00000100:00080000:1.0:1788575573.189619:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788575573.189620:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788575573.189627:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575573.189634:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575573.189647:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488914432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575573.189665:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575573/real 1788575573] req@ffff94d4049bd880 x1875450488914048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575573.189676:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575573.189681:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1788575573.189682:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd880 x1875450488914048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575573.189685:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575573.189694:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6721 00000100:00080000:0.0:1788575573.189698:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575573.189699:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575573.189703:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:1.0:1788575573.189708:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575573.189709:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488914048:192.168.202.125@tcp:250:kworker.0 00000008:00100000:1.0:1788575573.189717:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575573.189732:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488914560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575573.189736:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488914176:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575573.189757:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575573.189766:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575573/real 1788575573] req@ffff94d4049bca80 x1875450488914176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575573.189775:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488914176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575573.189780:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575573.189783:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575573.189785:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488914176:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575573.189788:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488914304:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575573.189801:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575573.189806:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575573/real 1788575573] req@ffff94d4049bd500 x1875450488914304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575573.189817:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488914304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575573.189821:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575573.189824:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575573.189826:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488914304:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575573.189829:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488914432:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575573.189841:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575573.189848:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575573/real 1788575573] req@ffff94d4049bdc00 x1875450488914432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575573.189855:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488914432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575573.189860:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575573.189861:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575573.189863:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488914432:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575573.189874:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488914560:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575573.189887:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575573.189893:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575573/real 1788575573] req@ffff94d4049be680 x1875450488914560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575573.189900:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488914560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575573.189909:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575573.189912:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575573.189915:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488914560:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575573.701319:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575574.149435:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575574.725295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575575.173393:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575575.749313:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575576.197411:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575576.773347:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575577.221419:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575577.797352:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575578.245412:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575578.309358:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575578.309373:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575578.309379:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575578.309392:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6726 00000100:00080000:3.0:1788575578.309407:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575578.309416:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575578.309468:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488914688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575578.309501:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575578.309505:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575578.309507:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575578.309511:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6726 00000100:00080000:3.0:1788575578.309515:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575578.309520:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575578.309531:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575578.309545:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488914688:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575578.309549:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488914816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575578.309562:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575578.309567:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575578.309569:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575578.309573:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6726 00000100:00080000:3.0:1788575578.309577:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575578.309582:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575578.309587:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575578.309600:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488914944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575578.309610:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575578.309614:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1788575578.309618:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575578.309618:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575578.309625:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6726 00000100:00080000:3.0:1788575578.309629:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575578.309634:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575578.309640:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575578.309652:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488915072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575578.309659:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575578/real 1788575578] req@ffff94d408c54e00 x1875450488914688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575578.309668:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575578.309673:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575578.309675:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575578.309680:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488914688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575578.309681:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6726 00000100:00080000:3.0:1788575578.309688:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575578.309694:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575578.309695:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00080000:0.0:1788575578.309701:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575578.309705:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488914688:192.168.202.125@tcp:250:kworker.0 00000008:00100000:3.0:1788575578.309705:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575578.309723:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488915200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575578.309735:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488914816:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575578.309753:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575578.309763:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575578/real 1788575578] req@ffff94d408c55c00 x1875450488914816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575578.309771:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488914816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575578.309776:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575578.309778:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575578.309786:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488914816:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575578.309790:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488914944:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575578.309802:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575578.309808:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575578/real 1788575578] req@ffff94d408c57b80 x1875450488914944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575578.309815:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488914944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575578.309820:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575578.309822:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575578.309824:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488914944:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575578.309828:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488915072:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575578.309846:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575578.309856:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575578/real 1788575578] req@ffff94d408c56d80 x1875450488915072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575578.309868:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488915072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575578.309876:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575578.309879:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575578.309882:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488915072:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575578.309898:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488915200:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575578.309917:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575578.309926:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575578/real 1788575578] req@ffff94d408c56680 x1875450488915200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575578.309934:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488915200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575578.309939:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575578.309941:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575578.309942:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488915200:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575578.821354:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575579.269367:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575579.845351:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575580.293434:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575580.869327:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575581.317404:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575581.893350:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575582.341401:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575582.917297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575583.365354:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575583.429321:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575583.429333:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575583.429338:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575583.429348:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6731 00000100:00080000:3.0:1788575583.429360:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575583.429366:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575583.429402:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488915328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575583.429423:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575583.429426:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575583.429427:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575583.429430:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6731 00000100:00080000:3.0:1788575583.429431:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575583.429434:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575583.429441:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575583.429448:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488915456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575583.429455:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575583.429458:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575583.429459:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575583.429463:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6731 00000100:00100000:0.0:1788575583.429464:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488915328:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575583.429465:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575583.429470:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575583.429474:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575583.429482:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488915584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575583.429496:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575583.429499:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575583.429501:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575583.429505:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6731 00000100:00080000:3.0:1788575583.429509:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575583.429513:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575583.429518:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788575583.429523:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575583.429531:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488915712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575583.429541:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575583.429544:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575583.429546:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575583.429551:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6731 00000100:00080000:3.0:1788575583.429555:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788575583.429556:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575583/real 1788575583] req@ffff94d408c57100 x1875450488915328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575583.429561:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575583.429567:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575583.429577:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488915328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575583.429578:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488915840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575583.429598:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575583.429603:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575583.429606:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488915328:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575583.429625:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488915456:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575583.429639:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575583.429646:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575583/real 1788575583] req@ffff94d408c54a80 x1875450488915456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575583.429657:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488915456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575583.429662:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575583.429664:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575583.429666:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488915456:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575583.429669:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488915584:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575583.429678:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575583.429683:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575583/real 1788575583] req@ffff94d408c56680 x1875450488915584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575583.429689:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488915584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575583.429694:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575583.429695:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575583.429697:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488915584:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575583.429700:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488915712:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575583.429710:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575583.429718:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575583/real 1788575583] req@ffff94d408c56d80 x1875450488915712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575583.429728:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488915712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575583.429741:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575583.429744:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575583.429747:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488915712:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575583.429753:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488915840:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575583.429772:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575583.429783:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575583/real 1788575583] req@ffff94d408c57b80 x1875450488915840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575583.429794:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488915840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575583.429801:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575583.429804:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575583.429807:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488915840:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575583.941329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575584.389339:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575584.965296:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575585.413379:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575585.989280:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575586.437323:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575587.013344:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575587.461378:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575588.037293:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575588.485355:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575588.549339:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575588.549349:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575588.549353:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575588.549363:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6736 00000100:00080000:1.0:1788575588.549378:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575588.549395:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575588.549431:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488915968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575588.549456:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575588.549460:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575588.549463:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575588.549467:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6736 00000100:00080000:1.0:1788575588.549471:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575588.549475:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575588.549484:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575588.549497:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488916096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575588.549500:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488915968:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788575588.549515:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575588.549518:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575588.549520:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575588.549525:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6736 00000100:00080000:1.0:1788575588.549527:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575588.549531:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575588.549536:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575588.549546:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488916224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575588.549555:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788575588.549556:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788575588.549561:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575588.549563:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575588.549568:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6736 00000100:00080000:1.0:1788575588.549571:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575588.549575:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575588.549579:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575588.549589:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575588/real 1788575588] req@ffff94d4049bf100 x1875450488915968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575588.549593:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488916352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575588.549607:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575588.549611:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488915968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575588.549611:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575588.549614:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575588.549621:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6736 00000100:00080000:1.0:1788575588.549626:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575588.549629:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575588.549633:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00080000:0.0:1788575588.549635:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575588.549640:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488915968:192.168.202.125@tcp:250:kworker.0 00000008:00100000:1.0:1788575588.549641:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575588.549655:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488916480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575588.549662:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488916096:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575588.549689:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575588.549703:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575588/real 1788575588] req@ffff94d4049be300 x1875450488916096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575588.549714:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488916096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575588.549719:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575588.549721:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575588.549723:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488916096:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575588.549726:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488916224:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575588.549740:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575588.549749:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575588/real 1788575588] req@ffff94d4049bf800 x1875450488916224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575588.549759:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488916224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575588.549764:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575588.549766:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575588.549768:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488916224:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575588.549771:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488916352:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575588.549784:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575588.549792:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575588/real 1788575588] req@ffff94d4049bed80 x1875450488916352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575588.549802:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488916352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575588.549809:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575588.549812:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575588.549815:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488916352:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575588.549829:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488916480:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575588.549849:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575588.549860:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575588/real 1788575588] req@ffff94d4049bc700 x1875450488916480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575588.549871:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488916480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575588.549879:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575588.549882:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575588.549885:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488916480:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575589.061343:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575589.509375:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575590.085296:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575590.533348:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575591.109338:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575591.557360:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575592.133320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575592.581328:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575593.157345:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575593.605366:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575593.669282:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575593.669295:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575593.669299:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575593.669309:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6741 00000100:00080000:1.0:1788575593.669323:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575593.669331:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575593.669366:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488916608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575593.669386:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575593.669389:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575593.669390:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575593.669394:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6741 00000100:00080000:1.0:1788575593.669396:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575593.669399:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575593.669407:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575593.669415:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488916736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575593.669422:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575593.669425:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575593.669428:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575593.669429:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488916608:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788575593.669431:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6741 00000100:00080000:1.0:1788575593.669436:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575593.669440:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575593.669443:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575593.669449:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488916864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575593.669455:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575593.669457:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575593.669459:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575593.669461:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6741 00000100:00080000:1.0:1788575593.669463:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575593.669465:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575593.669468:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575593.669474:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488916992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575593.669480:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575593.669486:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575593.669487:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575593.669489:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6741 00000100:00080000:1.0:1788575593.669491:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575593.669493:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575593.669496:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788575593.669504:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575593.669505:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488917120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575593.669540:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575593/real 1788575593] req@ffff94d4049bc000 x1875450488916608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575593.669564:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488916608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575593.669574:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575593.669578:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575593.669582:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488916608:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575593.669603:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488916736:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575593.669626:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575593.669639:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575593/real 1788575593] req@ffff94d4049bfb80 x1875450488916736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575593.669652:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488916736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575593.669660:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575593.669664:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575593.669667:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488916736:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575593.669687:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488916864:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575593.669707:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575593.669717:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575593/real 1788575593] req@ffff94d4049bd180 x1875450488916864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575593.669728:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488916864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575593.669742:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575593.669745:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575593.669748:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488916864:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575593.669754:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488916992:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575593.669783:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575593.669793:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575593/real 1788575593] req@ffff94d4049bce00 x1875450488916992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575593.669804:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488916992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575593.669821:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575593.669827:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575593.669830:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488916992:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575593.669835:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488917120:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575593.669850:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575593.669859:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575593/real 1788575593] req@ffff94d4049bea00 x1875450488917120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575593.669870:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488917120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575593.669877:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575593.669880:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575593.669883:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488917120:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575594.181322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575594.629342:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575595.205320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575595.653372:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575596.229335:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575596.677364:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575597.253312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575597.701377:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575598.277308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575598.725373:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575598.789296:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575598.789306:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575598.789310:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575598.789319:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6747 00000100:00080000:1.0:1788575598.789329:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575598.789336:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575598.789368:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488917248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575598.789397:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575598.789401:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575598.789402:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575598.789406:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6747 00000100:00080000:1.0:1788575598.789410:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575598.789415:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575598.789424:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575598.789438:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488917248:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575598.789439:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488917376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575598.789454:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575598.789458:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575598.789460:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575598.789463:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6747 00000100:00080000:1.0:1788575598.789467:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575598.789471:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575598.789476:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575598.789489:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488917504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575598.789490:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788575598.789501:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575598.789504:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575598.789506:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575598.789510:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6747 00000100:00080000:1.0:1788575598.789514:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575598.789517:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575598.789521:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575598.789528:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575598/real 1788575598] req@ffff94d4049bc380 x1875450488917248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575598.789531:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488917632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575598.789553:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575598.789556:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1788575598.789560:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc380 x1875450488917248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575598.789561:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575598.789569:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6747 00000100:00080000:1.0:1788575598.789575:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575598.789579:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575598.789581:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00080000:0.0:1788575598.789585:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575598.789590:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488917248:192.168.202.125@tcp:250:kworker.0 00000008:00100000:1.0:1788575598.789590:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575598.789608:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488917760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575598.789615:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488917376:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575598.789636:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575598.789646:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575598/real 1788575598] req@ffff94d4049bf480 x1875450488917376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575598.789653:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488917376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575598.789658:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575598.789660:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575598.789662:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488917376:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575598.789665:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488917504:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575598.789676:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575598.789681:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575598/real 1788575598] req@ffff94d4049bea00 x1875450488917504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575598.789690:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488917504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575598.789698:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575598.789700:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575598.789703:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488917504:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575598.789709:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488917632:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575598.789727:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575598.789737:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575598/real 1788575598] req@ffff94d4049bce00 x1875450488917632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575598.789745:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488917632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575598.789750:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575598.789751:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575598.789753:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488917632:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575598.789763:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488917760:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575598.789774:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575598.789780:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575598/real 1788575598] req@ffff94d4049bd180 x1875450488917760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575598.789791:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488917760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575598.789796:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575598.789799:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575598.789801:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488917760:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575599.301332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575599.749387:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575600.325322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575600.773391:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575601.349318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575601.797367:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575602.373506:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575602.821374:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575603.397331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575603.845427:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575603.909396:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575603.909407:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575603.909412:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575603.909426:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6752 00000100:00080000:3.0:1788575603.909441:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575603.909451:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575603.909518:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488917888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575603.909555:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575603.909559:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575603.909561:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575603.909565:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6752 00000100:00080000:3.0:1788575603.909569:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575603.909574:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575603.909585:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575603.909599:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488918016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575603.909610:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575603.909614:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575603.909616:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575603.909622:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6752 00000100:00100000:0.0:1788575603.909625:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488917888:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575603.909625:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575603.909631:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575603.909649:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575603.909664:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488918144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575603.909675:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575603.909679:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575603.909681:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575603.909685:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6752 00000100:00080000:3.0:1788575603.909689:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575603.909694:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575603.909699:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575603.909709:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488918272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575603.909718:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788575603.909724:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575603.909725:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575603.909729:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575603.909733:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6752 00000100:00080000:3.0:1788575603.909736:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575603.909739:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575603.909743:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575603.909752:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488918400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575603.909781:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575603/real 1788575603] req@ffff94d408c55c00 x1875450488917888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575603.909801:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488917888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575603.909811:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575603.909815:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575603.909819:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488917888:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575603.909826:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488918016:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575603.909847:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575603.909859:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575603/real 1788575603] req@ffff94d408c54e00 x1875450488918016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575603.909872:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488918016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575603.909881:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575603.909884:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575603.909887:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488918016:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575603.909914:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488918144:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575603.909935:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575603.909946:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575603/real 1788575603] req@ffff94d408c56a00 x1875450488918144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575603.909958:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488918144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575603.909966:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575603.909970:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575603.909973:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488918144:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575603.909978:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488918272:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575603.909999:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575603.910010:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575603/real 1788575603] req@ffff94d408c57800 x1875450488918272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575603.910022:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488918272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575603.910030:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575603.910033:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575603.910037:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488918272:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575603.910044:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488918400:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575603.910063:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575603.910074:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575603/real 1788575603] req@ffff94d408c54380 x1875450488918400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575603.910085:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488918400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575603.910093:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575603.910096:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575603.910100:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488918400:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575604.421335:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575604.869420:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575605.445333:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575605.893389:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575606.469336:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575606.917417:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575607.493332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575607.941344:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575608.517351:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575608.965391:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575609.029341:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575609.029352:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575609.029357:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575609.029368:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6757 00000100:00080000:3.0:1788575609.029378:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575609.029386:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575609.029421:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488918528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575609.029452:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575609.029456:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575609.029459:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575609.029463:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6757 00000100:00080000:3.0:1788575609.029468:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575609.029473:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575609.029478:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488918528:192.168.202.125@tcp:250:kworker.0 00000008:00100000:3.0:1788575609.029483:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575609.029501:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488918656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575609.029513:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575609.029517:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575609.029520:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575609.029524:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6757 00000100:00080000:3.0:1788575609.029529:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788575609.029533:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575609.029537:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575609.029544:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575609.029557:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488918784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575609.029571:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575609/real 1788575609] req@ffff94d408c55f80 x1875450488918528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575609.029573:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575609.029582:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575609.029588:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575609.029594:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6757 00000100:00100000:2.0:1788575609.029597:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488918528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575609.029600:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575609.029609:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575609.029609:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00080000:2.0:1788575609.029616:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000008:00100000:3.0:1788575609.029617:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575609.029620:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488918528:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575609.029634:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488918912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575609.029644:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488918656:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788575609.029647:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575609.029652:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575609.029655:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575609.029660:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6757 00000100:00080000:3.0:1788575609.029665:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788575609.029666:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575609.029673:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575609.029678:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575609/real 1788575609] req@ffff94d408c54700 x1875450488918656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788575609.029680:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575609.029692:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488918656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575609.029701:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788575609.029701:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488919040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575609.029706:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575609.029712:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488918656:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575609.029720:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488918784:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575609.029735:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575609.029743:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575609/real 1788575609] req@ffff94d408c56300 x1875450488918784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575609.029750:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488918784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575609.029755:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575609.029757:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575609.029760:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488918784:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575609.029769:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488918912:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575609.029782:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575609.029787:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575609/real 1788575609] req@ffff94d408c55880 x1875450488918912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575609.029794:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488918912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575609.029799:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575609.029801:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575609.029803:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488918912:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575609.029806:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488919040:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575609.029815:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575609.029820:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575609/real 1788575609] req@ffff94d408c55180 x1875450488919040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575609.029826:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488919040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575609.029831:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575609.029834:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575609.029835:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488919040:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575609.541294:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575609.989375:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575610.565326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575611.013405:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575611.589337:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575612.037383:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575612.613344:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575613.061461:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575613.637380:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575614.085413:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575614.149304:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575614.149311:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575614.149314:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575614.149321:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6762 00000100:00080000:2.0:1788575614.149335:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575614.149340:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575614.149369:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60700 x1875450488919168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575614.149402:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575614.149406:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575614.149408:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575614.149412:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6762 00000100:00080000:2.0:1788575614.149415:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575614.149419:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575614.149427:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575614.149436:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60380 x1875450488919296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575614.149443:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575614.149445:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575614.149446:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575614.149449:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6762 00000100:00080000:2.0:1788575614.149451:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788575614.149454:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488919168:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788575614.149455:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575614.149461:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575614.149467:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62300 x1875450488919424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575614.149474:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575614.149476:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575614.149477:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575614.149480:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6762 00000100:00080000:2.0:1788575614.149482:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575614.149484:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575614.149487:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575614.149495:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488919552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575614.149500:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575614.149502:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575614.149503:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575614.149505:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6762 00000100:00080000:2.0:1788575614.149508:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575614.149510:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575614.149512:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575614.149519:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488919680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575614.149548:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488919296:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575614.149588:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488919424:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575614.149605:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488919552:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575614.149621:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488919680:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575614.661357:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575615.109451:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575615.685316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575616.133403:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575616.709302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575617.157401:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788575617.221499:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:1.0:1788575617.221547:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788575617.221562:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575617.221607:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575617.221616:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575617.221624:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575617.221631:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:0.0:1788575617.221717:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575614/real 1788575617] req@ffff94d408c62300 x1875450488919424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575617.221746:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62300 x1875450488919424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575617.221759:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575617.221763:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575617.221767:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488919424:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575617.221774:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575614/real 1788575617] req@ffff94d408c62680 x1875450488919552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575617.221787:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488919552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575617.221793:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575617.221795:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575617.221800:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488919552:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575617.221805:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575614/real 1788575617] req@ffff94d408c63800 x1875450488919680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575617.221815:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488919680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575617.221822:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575617.221825:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575617.221828:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488919680:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575617.221834:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575614/real 1788575617] req@ffff94d408c60700 x1875450488919168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575617.221844:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60700 x1875450488919168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575617.221851:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575617.221854:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575617.221856:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488919168:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575617.221861:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575614/real 1788575617] req@ffff94d408c60380 x1875450488919296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575617.221870:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60380 x1875450488919296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575617.221876:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575617.221879:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575617.221883:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488919296:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575617.733330:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575618.181403:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575618.757330:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575619.205438:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575619.269347:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575619.269356:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575619.269360:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575619.269368:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6767 00000100:00080000:2.0:1788575619.269377:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575619.269384:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575619.269410:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488919808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575619.269438:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575619.269442:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575619.269444:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575619.269448:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6767 00000100:00080000:2.0:1788575619.269452:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575619.269455:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575619.269464:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575619.269477:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488919936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575619.269487:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575619.269490:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575619.269492:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575619.269495:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6767 00000100:00080000:2.0:1788575619.269497:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575619.269500:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575619.269503:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575619.269511:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488920064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575619.269517:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575619.269518:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575619.269520:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575619.269523:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6767 00000100:00080000:2.0:1788575619.269526:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575619.269529:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575619.269533:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575619.269542:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488920192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575619.269550:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575619.269553:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575619.269555:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575619.269558:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6767 00000100:00080000:2.0:1788575619.269560:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575619.269564:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575619.269568:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575619.269576:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488920320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575619.269662:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488919808:192.168.202.125@tcp:250:kworker.0 00000800:00000100:0.0:1788575619.269720:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575619.269752:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575619/real 1788575619] req@ffff94d408c60000 x1875450488919808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575619.269768:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488919808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575619.269779:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575619.269783:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575619.269787:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488919808:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575619.269793:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488919936:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575619.269814:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575619.269826:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575619/real 1788575619] req@ffff94d408c62d80 x1875450488919936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575619.269836:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488919936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575619.269844:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575619.269847:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575619.269849:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488919936:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575619.269855:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488920064:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575619.269872:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575619.269881:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575619/real 1788575619] req@ffff94d408c61180 x1875450488920064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575619.269892:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488920064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575619.269900:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575619.269902:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575619.269905:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488920064:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575619.269911:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488920192:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575619.269926:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575619.269946:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575619/real 1788575619] req@ffff94d408c63480 x1875450488920192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575619.269957:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488920192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575619.269964:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575619.269967:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575619.269970:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488920192:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575619.269975:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488920320:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575619.269991:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575619.270000:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575619/real 1788575619] req@ffff94d408c61f80 x1875450488920320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575619.270011:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488920320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575619.270019:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575619.270021:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575619.270024:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488920320:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575619.781366:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575620.229394:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575620.807336:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575621.253471:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575621.829362:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575622.277342:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575622.853344:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575623.301386:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575623.878438:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575624.325417:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575624.389373:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575624.389381:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575624.389384:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575624.389393:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6772 00000100:00080000:3.0:1788575624.389405:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575624.389412:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575624.389448:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488920448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575624.389477:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575624.389480:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575624.389482:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575624.389486:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6772 00000100:00080000:3.0:1788575624.389489:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575624.389500:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575624.389510:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575624.389523:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488920576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575624.389528:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488920448:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575624.389541:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575624.389545:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575624.389546:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575624.389551:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6772 00000100:00080000:3.0:1788575624.389555:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575624.389559:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575624.389563:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575624.389576:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488920704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575624.389585:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575624.389593:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575624.389596:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575624.389600:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6772 00000100:00080000:3.0:1788575624.389603:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788575624.389605:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575624.389610:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575624.389616:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575624.389629:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488920832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575624.389642:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575624.389645:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575624/real 1788575624] req@ffff94d408c57480 x1875450488920448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575624.389646:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575624.389650:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575624.389659:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6772 00000100:00080000:3.0:1788575624.389665:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575624.389670:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575624.389674:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488920448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788575624.389677:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788575624.389690:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788575624.389692:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488920960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575624.389698:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575624.389703:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488920448:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575624.389732:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488920576:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575624.389761:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575624.389773:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575624/real 1788575624] req@ffff94d408c54000 x1875450488920576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575624.389785:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488920576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575624.389793:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575624.389796:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575624.389800:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488920576:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575624.389805:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488920704:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575624.389822:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575624.389832:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575624/real 1788575624] req@ffff94d408c55180 x1875450488920704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575624.389857:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488920704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575624.389866:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575624.389869:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575624.389873:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488920704:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575624.389879:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488920832:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575624.389897:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575624.389908:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575624/real 1788575624] req@ffff94d408c55880 x1875450488920832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575624.389919:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488920832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575624.389927:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575624.389930:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575624.389934:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488920832:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575624.389939:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488920960:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575624.389960:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575624.389969:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575624/real 1788575624] req@ffff94d408c56300 x1875450488920960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575624.389979:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488920960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575624.389987:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575624.389990:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575624.389993:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488920960:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575624.902660:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575625.349408:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575625.926078:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575626.373388:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575626.949367:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575627.397443:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575627.973325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575628.421424:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575628.997350:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788575629.445407:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575629.510073:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575629.510088:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575629.510093:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575629.510106:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6777 00000100:00080000:3.0:1788575629.510120:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575629.510129:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575629.510178:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488921088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575629.510230:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575629.510234:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575629.510237:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575629.510242:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6777 00000100:00080000:3.0:1788575629.510245:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575629.510250:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575629.510261:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575629.510264:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488921088:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575629.510278:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488921216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575629.510304:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575629.510308:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575629.510311:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575629.510315:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6777 00000100:00080000:3.0:1788575629.510319:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575629.510324:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575629.510329:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575629.510341:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488921344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575629.510352:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788575629.510353:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575629.510359:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575629.510361:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575629.510366:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6777 00000100:00080000:3.0:1788575629.510371:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575629.510376:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575629.510382:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575629.510396:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488921472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575629.510401:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575629/real 1788575629] req@ffff94d408c54700 x1875450488921088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575629.510421:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575629.510422:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488921088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575629.510429:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575629.510434:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575629.510442:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6777 00000100:00080000:0.0:1788575629.510443:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575629.510446:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575629.510451:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:3.0:1788575629.510454:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575629.510460:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488921088:192.168.202.125@tcp:250:kworker.0 00000008:00100000:3.0:1788575629.510463:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575629.510479:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488921600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575629.510501:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488921216:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575629.510527:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575629.510542:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575629/real 1788575629] req@ffff94d408c55f80 x1875450488921216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575629.510554:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488921216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575629.510562:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575629.510565:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575629.510568:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488921216:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575629.510574:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488921344:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575629.510605:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575629.510616:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575629/real 1788575629] req@ffff94d408c54380 x1875450488921344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575629.510633:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488921344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575629.510641:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575629.510644:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575629.510647:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488921344:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575629.510653:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488921472:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575629.510672:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575629.510683:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575629/real 1788575629] req@ffff94d408c57800 x1875450488921472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575629.510694:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488921472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575629.510701:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575629.510704:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575629.510707:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488921472:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575629.510713:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488921600:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575629.510730:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575629.510739:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575629/real 1788575629] req@ffff94d408c56a00 x1875450488921600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575629.510750:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488921600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575629.510762:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575629.510765:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575629.510768:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488921600:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575630.021340:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575630.469412:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575631.045329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575631.493397:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575632.069344:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575632.517524:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575633.093376:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575633.541503:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575634.121490:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575634.565444:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575634.629385:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575634.629400:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575634.629405:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575634.629418:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6782 00000100:00080000:1.0:1788575634.629432:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575634.629442:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575634.629492:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488921728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575634.629527:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575634.629532:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575634.629533:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575634.629537:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6782 00000100:00080000:1.0:1788575634.629541:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575634.629545:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575634.629556:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575634.629569:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488921856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575634.629579:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488921728:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788575634.629590:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575634.629594:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575634.629597:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575634.629601:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6782 00000100:00080000:1.0:1788575634.629605:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575634.629609:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575634.629615:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575634.629627:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488921984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575634.629642:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575634.629645:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575634.629647:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575634.629652:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6782 00000100:00080000:1.0:1788575634.629656:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575634.629659:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575634.629665:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788575634.629671:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575634.629678:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488922112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575634.629689:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575634.629692:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575634.629694:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575634.629699:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6782 00000100:00080000:1.0:1788575634.629703:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575634.629707:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575634.629712:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575634.629723:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575634/real 1788575634] req@ffff94d4049bfb80 x1875450488921728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575634.629726:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488922240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575634.629756:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488921728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575634.629765:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575634.629769:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575634.629773:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488921728:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575634.629800:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488921856:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575634.629825:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575634.629838:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575634/real 1788575634] req@ffff94d4049bc000 x1875450488921856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575634.629847:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488921856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575634.629857:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575634.629860:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575634.629862:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488921856:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575634.629866:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488921984:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575634.629887:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575634.629894:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575634/real 1788575634] req@ffff94d4049bc700 x1875450488921984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575634.629902:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488921984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575634.629908:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575634.629910:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575634.629912:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488921984:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575634.629915:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488922112:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575634.629926:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575634.629932:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575634/real 1788575634] req@ffff94d4049bed80 x1875450488922112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575634.629938:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488922112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575634.629942:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575634.629944:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575634.629947:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488922112:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575634.629950:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488922240:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575634.629963:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575634.629973:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575634/real 1788575634] req@ffff94d4049bf800 x1875450488922240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575634.629983:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488922240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575634.629990:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575634.629992:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575634.629994:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488922240:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575635.141356:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575635.589423:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575636.165336:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575636.613415:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575637.189329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575637.637403:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575638.213355:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575638.661415:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575639.237346:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575639.685393:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575639.749304:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575639.749321:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575639.749325:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575639.749334:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6788 00000100:00080000:3.0:1788575639.749344:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575639.749351:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575639.749384:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488922368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575639.749416:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575639.749419:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575639.749421:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575639.749425:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6788 00000100:00080000:3.0:1788575639.749429:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575639.749433:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575639.749442:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575639.749465:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488922496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575639.749471:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488922368:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575639.749479:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575639.749484:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575639.749486:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575639.749489:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6788 00000100:00080000:3.0:1788575639.749493:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575639.749497:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575639.749502:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575639.749515:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488922624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575639.749525:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575639.749528:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575639.749530:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575639.749534:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6788 00000800:00000100:0.0:1788575639.749539:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575639.749551:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575639.749558:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575639.749564:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575639.749577:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575639/real 1788575639] req@ffff94d408c54e00 x1875450488922368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575639.749579:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488922752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575639.749603:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575639.749606:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488922368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575639.749608:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575639.749612:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575639.749619:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575639.749620:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6788 00000100:00080000:0.0:1788575639.749624:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:3.0:1788575639.749627:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788575639.749630:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488922368:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575639.749635:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575639.749645:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575639.749658:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488922880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575639.749665:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488922496:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575639.749689:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575639.749698:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575639/real 1788575639] req@ffff94d408c55c00 x1875450488922496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575639.749706:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488922496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575639.749711:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575639.749714:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575639.749716:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488922496:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575639.749719:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488922624:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575639.749728:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575639.749734:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575639/real 1788575639] req@ffff94d408c57b80 x1875450488922624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575639.749740:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488922624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575639.749745:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575639.749747:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575639.749748:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488922624:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575639.749751:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488922752:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575639.749764:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575639.749774:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575639/real 1788575639] req@ffff94d408c56d80 x1875450488922752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575639.749785:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488922752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575639.749792:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575639.749795:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575639.749797:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488922752:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575639.749810:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488922880:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575639.749827:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575639.749837:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575639/real 1788575639] req@ffff94d408c56680 x1875450488922880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575639.749847:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488922880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575639.749855:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575639.749858:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575639.749861:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488922880:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575640.261353:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575640.709432:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575641.285305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575641.733372:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575642.309311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575642.757430:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575643.333358:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575643.781526:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575644.357365:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575644.805360:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575644.869345:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575644.869359:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575644.869363:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575644.869375:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6793 00000100:00080000:3.0:1788575644.869386:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575644.869395:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575644.869438:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488923008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575644.869471:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575644.869474:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575644.869475:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575644.869478:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6793 00000100:00080000:3.0:1788575644.869480:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575644.869482:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575644.869491:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575644.869505:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488923136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575644.869517:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575644.869521:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788575644.869526:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488923008:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575644.869526:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575644.869536:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6793 00000100:00080000:3.0:1788575644.869542:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575644.869547:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575644.869554:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575644.869567:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488923264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575644.869576:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575644.869580:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575644.869582:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575644.869587:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6793 00000100:00080000:3.0:1788575644.869592:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575644.869598:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:1.0:1788575644.869602:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575644.869648:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575644/real 1788575644] req@ffff94d408c54a80 x1875450488923008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788575644.869669:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575644.869672:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488923008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575644.869686:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788575644.869687:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488923392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575644.869697:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:3.0:1788575644.869704:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575644.869706:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488923008:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575644.869736:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488923136:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575644.869780:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575644.869794:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575644/real 1788575644] req@ffff94d408c57100 x1875450488923136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575644.869808:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488923136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575644.869816:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575644.869820:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575644.869823:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488923136:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575644.869829:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488923264:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788575644.869837:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575644.869841:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788575644.869858:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575644.869859:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6793 00000100:00080000:3.0:1788575644.869872:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575644.869877:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575644/real 1788575644] req@ffff94d408c56680 x1875450488923264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575644.869881:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575644.869892:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575644.869899:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488923264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575644.869919:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575644.869923:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575644.869926:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488923264:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575644.869932:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488923392:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575644.869958:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575644.869973:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575644/real 1788575644] req@ffff94d408c56d80 x1875450488923392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575644.869980:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488923520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575644.869990:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488923392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575644.870001:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575644.870005:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575644.870009:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488923392:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575644.870028:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488923520:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575644.870046:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575644.870054:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575644/real 1788575644] req@ffff94d408c57b80 x1875450488923520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575644.870066:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488923520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575644.870071:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575644.870073:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575644.870075:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488923520:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575645.381388:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575645.829416:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575646.405347:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575646.853380:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575647.429321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575647.877403:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575648.453317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575648.901386:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575649.477308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575649.925397:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575649.989384:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575649.989396:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575649.989400:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575649.989410:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6798 00000100:00080000:0.0:1788575649.989420:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575649.989427:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575649.989466:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488923648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575649.989492:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575649.989496:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575649.989498:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575649.989502:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6798 00000100:00080000:0.0:1788575649.989505:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575649.989509:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575649.989517:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575649.989531:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488923776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575649.989541:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575649.989545:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575649.989547:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575649.989551:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6798 00000100:00080000:0.0:1788575649.989554:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575649.989558:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575649.989564:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575649.989575:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488923648:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575649.989581:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488923904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575649.989596:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575649.989599:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575649.989601:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575649.989605:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6798 00000100:00080000:0.0:1788575649.989609:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575649.989613:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575649.989619:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575649.989631:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488924032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575649.989640:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575649.989643:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575649.989645:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575649.989649:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6798 00000100:00080000:0.0:1788575649.989663:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575649.989669:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575649.989674:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788575649.989683:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575649.989690:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488924160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575649.989748:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575649/real 1788575649] req@ffff94d408d3ea00 x1875450488923648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575649.989769:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488923648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575649.989781:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575649.989785:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575649.989789:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488923648:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575649.989798:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488923776:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575649.989822:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575649.989833:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575649/real 1788575649] req@ffff94d408d3ca80 x1875450488923776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575649.989841:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488923776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575649.989853:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575649.989856:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575649.989860:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488923776:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575649.989888:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488923904:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575649.989910:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575649.989922:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575649/real 1788575649] req@ffff94d408d3d180 x1875450488923904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575649.989933:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488923904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575649.989941:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575649.989945:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575649.989948:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488923904:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575649.989954:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488924032:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575649.989973:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575649.989983:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575649/real 1788575649] req@ffff94d408d3df80 x1875450488924032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575649.989994:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488924032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575649.990002:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575649.990005:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575649.990008:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488924032:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575649.990015:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488924160:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575649.990035:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575649.990054:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575649/real 1788575649] req@ffff94d408d3ce00 x1875450488924160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575649.990071:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488924160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575649.990079:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575649.990082:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575649.990085:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488924160:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575650.501293:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575650.949357:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575651.525318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575651.973398:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575652.549303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575652.997443:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575653.573305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575654.021369:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575654.597323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575655.045389:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575655.109318:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575655.109327:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575655.109330:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575655.109340:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6803 00000100:00080000:2.0:1788575655.109364:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575655.109372:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575655.109406:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488924288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575655.109433:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575655.109437:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575655.109438:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575655.109442:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6803 00000100:00080000:2.0:1788575655.109445:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575655.109450:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575655.109459:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575655.109466:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488924288:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575655.109473:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488924416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575655.109497:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575655.109501:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575655.109503:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575655.109507:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6803 00000100:00080000:2.0:1788575655.109511:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575655.109515:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575655.109521:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575655.109525:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575655.109546:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488924544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575655.109554:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575655/real 1788575655] req@ffff94d408c62a00 x1875450488924288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575655.109562:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575655.109566:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575655.109569:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575655.109575:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488924288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575655.109575:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6803 00000100:00080000:2.0:1788575655.109582:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575655.109590:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00080000:1.0:1788575655.109593:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1788575655.109596:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788575655.109600:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575655.109606:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488924288:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575655.109613:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488924672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575655.109623:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575655.109625:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575655.109627:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575655.109637:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488924416:192.168.202.125@tcp:38:kworker.0 00000100:00080000:2.0:1788575655.109642:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6803 00000100:00080000:2.0:1788575655.109647:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575655.109652:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575655.109657:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788575655.109662:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575655.109672:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488924800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575655.109678:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575655/real 1788575655] req@ffff94d408c63b80 x1875450488924416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575655.109696:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488924416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575655.109708:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575655.109710:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575655.109712:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488924416:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575655.109716:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488924544:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575655.109730:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575655.109737:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575655/real 1788575655] req@ffff94d408c60a80 x1875450488924544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575655.109744:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488924544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575655.109751:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575655.109753:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575655.109757:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488924544:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575655.109761:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488924672:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575655.109778:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575655.109788:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575655/real 1788575655] req@ffff94d408c61500 x1875450488924672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575655.109799:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488924672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575655.109805:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575655.109807:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575655.109809:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488924672:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575655.109826:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488924800:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575655.109839:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575655.109845:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575655/real 1788575655] req@ffff94d408c61880 x1875450488924800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575655.109852:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488924800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575655.109857:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575655.109859:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575655.109860:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488924800:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575655.621284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575656.069393:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575656.645337:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575657.093382:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575657.669317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575658.117518:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575658.693281:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575659.141399:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575659.717676:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575660.165396:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575660.230079:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575660.230089:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575660.230093:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575660.230103:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6808 00000100:00080000:1.0:1788575660.230112:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575660.230119:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575660.230153:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488924928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575660.230179:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575660.230183:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575660.230185:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575660.230189:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6808 00000100:00080000:1.0:1788575660.230192:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575660.230196:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575660.230234:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488924928:192.168.202.125@tcp:250:kworker.0 00000008:00100000:1.0:1788575660.230239:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575660.230256:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488925056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575660.230267:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575660.230270:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575660.230273:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575660.230276:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6808 00000100:00080000:1.0:1788575660.230281:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575660.230286:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575660.230291:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788575660.230296:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575660.230304:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488925184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575660.230314:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575660.230317:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1788575660.230335:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575660/real 1788575660] req@ffff94d4049be300 x1875450488924928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575660.230364:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488924928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575660.230381:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575660.230386:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575660.230390:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488924928:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575660.230412:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488925056:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575660.230435:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575660.230448:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575660/real 1788575660] req@ffff94d4049bf100 x1875450488925056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575660.230460:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488925056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575660.230468:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575660.230472:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575660.230473:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575660.230481:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488925056:192.168.202.125@tcp:38:kworker.0 00000100:00080000:1.0:1788575660.230483:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6808 00000100:00100000:0.0:1788575660.230492:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488925184:192.168.202.125@tcp:38:kworker.0 00000100:00080000:1.0:1788575660.230493:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575660.230502:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575660.230510:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788575660.230520:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575660.230526:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488925312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575660.230535:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575660/real 1788575660] req@ffff94d4049be680 x1875450488925184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575660.230541:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575660.230546:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1788575660.230547:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488925184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575660.230551:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575660.230559:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575660.230561:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6808 00000100:00080000:0.0:1788575660.230564:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575660.230568:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488925184:192.168.202.125@tcp:38:kworker.0 00000100:00080000:1.0:1788575660.230570:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575660.230576:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575660.230581:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575660.230590:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488925312:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575660.230595:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488925440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575660.230608:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575660.230618:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575660/real 1788575660] req@ffff94d4049bdc00 x1875450488925312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575660.230627:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488925312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575660.230631:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575660.230633:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575660.230635:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488925312:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575660.230641:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488925440:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575660.230650:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575660.230655:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575660/real 1788575660] req@ffff94d4049bd500 x1875450488925440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575660.230661:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488925440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575660.230669:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575660.230671:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575660.230673:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488925440:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575660.741577:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575661.189406:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575661.765356:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575662.213390:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575662.790093:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575663.237395:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575663.813306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575664.261428:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575664.837325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575665.285647:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575665.349404:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575665.349417:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575665.349422:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575665.349434:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6813 00000100:00080000:2.0:1788575665.349449:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575665.349457:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575665.349502:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488925568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575665.349533:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575665.349536:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575665.349538:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575665.349542:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6813 00000100:00080000:2.0:1788575665.349545:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575665.349548:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575665.349559:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575665.349571:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60e00 x1875450488925696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575665.349580:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575665.349584:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575665.349586:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575665.349591:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6813 00000100:00100000:0.0:1788575665.349595:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488925568:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788575665.349595:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575665.349601:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575665.349608:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575665.349618:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63100 x1875450488925824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575665.349628:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575665.349631:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575665.349633:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575665.349644:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6813 00000100:00080000:2.0:1788575665.349651:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575665.349655:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575665.349660:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575665.349672:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488925952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575665.349682:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575665.349686:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1788575665.349688:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788575665.349689:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575665.349696:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6813 00000100:00080000:2.0:1788575665.349700:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575665.349703:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575665.349707:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575665.349718:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488926080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575665.349750:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575665/real 1788575665] req@ffff94d408c61c00 x1875450488925568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575665.349773:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488925568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575665.349784:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575665.349788:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575665.349793:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488925568:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575665.349802:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488925696:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575665.349830:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575665.349843:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575665/real 1788575665] req@ffff94d408c60e00 x1875450488925696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575665.349856:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60e00 x1875450488925696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575665.349871:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575665.349874:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575665.349878:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488925696:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575665.349908:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488925824:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575665.349930:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575665.349941:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575665/real 1788575665] req@ffff94d408c63100 x1875450488925824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575665.349952:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63100 x1875450488925824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575665.349961:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575665.349964:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575665.349967:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488925824:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575665.349973:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488925952:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575665.349994:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575665.350005:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575665/real 1788575665] req@ffff94d408c61880 x1875450488925952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575665.350017:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488925952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575665.350024:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575665.350028:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575665.350032:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488925952:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575665.350038:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488926080:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575665.350059:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575665.350078:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575665/real 1788575665] req@ffff94d408c61500 x1875450488926080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575665.350090:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488926080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575665.350098:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575665.350101:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575665.350105:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488926080:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575665.861296:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575666.309687:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575666.885328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575667.333389:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575667.909286:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575668.357377:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575668.933312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575669.381412:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575669.957324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575670.405451:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575670.469321:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575670.469329:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575670.469332:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575670.469340:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6818 00000100:00080000:0.0:1788575670.469361:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575670.469367:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575670.469404:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488926208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575670.469425:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575670.469428:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575670.469430:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575670.469439:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6818 00000100:00080000:0.0:1788575670.469442:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575670.469446:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575670.469455:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575670.469469:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488926336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575670.469472:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488926208:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788575670.469485:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575670.469488:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575670.469489:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575670.469492:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6818 00000100:00080000:0.0:1788575670.469495:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575670.469498:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575670.469502:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575670.469510:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ed80 x1875450488926464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575670.469516:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575670.469518:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575670.469519:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575670.469523:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6818 00000100:00080000:0.0:1788575670.469526:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575670.469529:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:1.0:1788575670.469534:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:0.0:1788575670.469536:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575670.469551:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488926592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575670.469561:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575670.469564:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575670.469566:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575670.469568:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575670/real 1788575670] req@ffff94d408d3f480 x1875450488926208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575670.469573:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6818 00000100:00080000:0.0:1788575670.469579:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575670.469582:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575670.469589:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575670.469596:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488926208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575670.469600:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488926720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575670.469615:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575670.469619:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575670.469623:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488926208:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575670.469675:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488926336:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575670.469701:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575670.469714:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575670/real 1788575670] req@ffff94d408d3c700 x1875450488926336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575670.469725:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488926336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575670.469732:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575670.469734:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575670.469736:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488926336:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575670.469740:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488926464:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575670.469759:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575670.469765:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575670/real 1788575670] req@ffff94d408d3ed80 x1875450488926464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575670.469772:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ed80 x1875450488926464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575670.469780:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575670.469782:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575670.469784:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488926464:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575670.469788:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488926592:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575670.469798:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575670.469803:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575670/real 1788575670] req@ffff94d408d3c380 x1875450488926592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575670.469814:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488926592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575670.469819:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575670.469820:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575670.469822:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488926592:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575670.469825:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488926720:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575670.469833:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575670.469838:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575670/real 1788575670] req@ffff94d408d3d880 x1875450488926720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575670.469844:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488926720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575670.469848:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575670.469850:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575670.469851:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488926720:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575670.981304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575671.429377:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575672.005322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575672.453379:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575673.029317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575673.477447:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575674.053332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575674.501402:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575675.077317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575675.525402:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575675.589387:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575675.589401:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575675.589407:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575675.589420:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6823 00000100:00080000:0.0:1788575675.589435:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575675.589444:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575675.589491:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488926848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575675.589527:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575675.589532:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575675.589534:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575675.589539:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6823 00000100:00080000:0.0:1788575675.589542:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575675.589546:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575675.589557:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575675.589571:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488926976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575675.589582:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575675.589586:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575675.589588:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575675.589592:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6823 00000100:00080000:0.0:1788575675.589596:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575675.589601:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575675.589607:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575675.589622:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488926848:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575675.589625:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488927104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575675.589649:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575675.589658:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575675.589661:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575675.589667:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6823 00000100:00080000:0.0:1788575675.589673:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575675.589678:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575675.589684:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575675.589699:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488927232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575675.589710:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788575675.589713:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788575675.589716:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575675.589719:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575675.589725:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6823 00000100:00080000:0.0:1788575675.589730:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575675.589736:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575675.589741:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575675.589756:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ed80 x1875450488927360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575675.589766:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575675/real 1788575675] req@ffff94d408d3e300 x1875450488926848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575675.589789:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488926848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575675.589805:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575675.589810:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575675.589814:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488926848:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575675.589822:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488926976:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575675.589848:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575675.589860:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575675/real 1788575675] req@ffff94d408d3dc00 x1875450488926976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575675.589871:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488926976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575675.589878:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575675.589881:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575675.589884:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488926976:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575675.589908:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488927104:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575675.589927:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575675.589939:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575675/real 1788575675] req@ffff94d408d3d880 x1875450488927104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575675.589966:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488927104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575675.589975:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575675.589978:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575675.589981:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488927104:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575675.589988:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488927232:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575675.590016:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575675.590027:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575675/real 1788575675] req@ffff94d408d3c380 x1875450488927232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575675.590037:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488927232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575675.590042:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575675.590044:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575675.590046:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488927232:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575675.590050:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488927360:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575675.590063:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575675.590069:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575675/real 1788575675] req@ffff94d408d3ed80 x1875450488927360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575675.590075:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ed80 x1875450488927360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575675.590079:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575675.590081:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575675.590083:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488927360:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575676.101314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575676.549385:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575677.125316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575677.573389:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575678.149290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575678.597446:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575679.173312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575679.621537:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575680.197297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575680.645329:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575680.709401:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575680.709410:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575680.709415:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575680.709427:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6829 00000100:00080000:3.0:1788575680.709447:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575680.709454:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575680.709494:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488927488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575680.709588:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488927488:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575680.709604:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575680.709608:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575680.709610:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575680.709614:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6829 00000100:00080000:3.0:1788575680.709619:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575680.709623:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575680.709633:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575680.709647:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488927616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575680.709668:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575680.709673:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575680.709675:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575680.709681:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6829 00000100:00080000:3.0:1788575680.709687:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788575680.709701:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488927616:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788575680.709704:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575680.709712:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575680.709730:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488927744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575680.709818:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575680.709822:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575680.709831:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575680.709836:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6829 00000100:00100000:2.0:1788575680.709862:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488927744:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788575680.709976:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575680.709981:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575680.709986:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575680.710010:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488927872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575680.710032:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575680.710036:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575680.710038:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575680.710042:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6829 00000100:00080000:3.0:1788575680.710051:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575680.710056:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575680.710062:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575680.710074:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488928000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575680.710240:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488927872:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575680.710271:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488928000:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575681.221317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575681.669403:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575682.245269:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575682.693391:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575683.269313:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575683.718231:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788575683.781503:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:1.0:1788575683.781549:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788575683.781563:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575683.781597:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575683.781603:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575683.781619:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575683.781626:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:2.0:1788575683.781686:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575680/real 1788575683] req@ffff94d408c57800 x1875450488927872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575683.781710:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488927872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575683.781721:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575683.781725:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575683.781729:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488927872:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575683.781737:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575680/real 1788575683] req@ffff94d408c54380 x1875450488928000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575683.781748:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488928000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575683.781755:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575683.781758:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575683.781761:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488928000:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575683.781768:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575680/real 1788575683] req@ffff94d408c56a00 x1875450488927744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575683.781778:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488927744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575683.781785:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575683.781788:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575683.781791:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488927744:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575683.781798:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575680/real 1788575683] req@ffff94d408c54e00 x1875450488927616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575683.781807:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488927616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575683.781814:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575683.781817:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575683.781820:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488927616:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575683.781826:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575680/real 1788575683] req@ffff94d408c55c00 x1875450488927488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575683.781837:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488927488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575683.781844:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575683.781847:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575683.781850:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488927488:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575684.294458:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575684.741389:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575685.317321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575685.765372:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575685.829348:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575685.829360:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575685.829364:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575685.829374:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6834 00000100:00080000:0.0:1788575685.829383:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575685.829389:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575685.829422:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488928128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575685.829454:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575685.829458:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575685.829460:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575685.829464:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6834 00000100:00080000:0.0:1788575685.829469:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575685.829473:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575685.829481:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575685.829494:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488928256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575685.829504:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575685.829507:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575685.829509:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575685.829512:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6834 00000100:00080000:0.0:1788575685.829515:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575685.829519:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575685.829523:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575685.829532:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488928384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575685.829541:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575685.829544:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575685.829546:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575685.829549:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6834 00000100:00100000:2.0:1788575685.829557:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488928128:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788575685.829559:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575685.829568:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575685.829573:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575685.829586:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488928512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575685.829597:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575685.829600:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575685.829602:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575685.829606:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6834 00000100:00080000:0.0:1788575685.829609:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788575685.829623:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788575685.829628:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575685.829634:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575685.829647:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488928640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575685.829664:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575685/real 1788575685] req@ffff94d408d3c700 x1875450488928128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575685.829684:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488928128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575685.829693:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575685.829697:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575685.829701:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488928128:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575685.829707:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488928256:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575685.829728:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575685.829740:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575685/real 1788575685] req@ffff94d408d3f480 x1875450488928256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575685.829758:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488928256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575685.829764:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575685.829772:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575685.829775:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488928256:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575685.829780:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488928384:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575685.829799:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575685.829809:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575685/real 1788575685] req@ffff94d408d3ce00 x1875450488928384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575685.829819:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488928384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575685.829826:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575685.829829:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575685.829832:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488928384:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575685.829857:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488928512:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575685.829875:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575685.829884:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575685/real 1788575685] req@ffff94d408d3df80 x1875450488928512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575685.829894:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488928512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575685.829905:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575685.829908:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575685.829910:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488928512:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575685.829915:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488928640:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575685.829946:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575685.829957:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575685/real 1788575685] req@ffff94d408d3d180 x1875450488928640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575685.829968:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488928640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575685.829975:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575685.829978:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575685.829980:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488928640:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575686.341326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575686.789537:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575687.365314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575687.813386:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575688.389306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575688.837386:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575689.414156:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575689.861385:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575690.438387:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575690.885391:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575690.949331:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575690.949350:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575690.949353:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575690.949362:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6839 00000100:00080000:1.0:1788575690.949371:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575690.949376:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575690.949414:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488928768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575690.949441:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575690.949444:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575690.949445:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575690.949448:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6839 00000100:00080000:1.0:1788575690.949451:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575690.949453:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575690.949460:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575690.949469:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd880 x1875450488928896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575690.949476:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575690.949478:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575690.949479:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575690.949482:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6839 00000100:00080000:1.0:1788575690.949485:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575690.949489:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575690.949494:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488928768:192.168.202.125@tcp:250:kworker.0 00000008:00100000:1.0:1788575690.949495:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575690.949511:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488929024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575690.949521:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575690.949525:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575690.949527:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575690.949530:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6839 00000100:00080000:1.0:1788575690.949535:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575690.949539:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575690.949543:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575690.949552:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488929152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575690.949557:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575690.949561:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575690.949563:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788575690.949563:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788575690.949569:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6839 00000100:00080000:1.0:1788575690.949572:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575690.949574:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575690.949578:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575690.949585:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488929280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575690.949603:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575690/real 1788575690] req@ffff94d4049bca80 x1875450488928768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575690.949617:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488928768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575690.949624:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575690.949627:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575690.949629:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488928768:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575690.949634:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488928896:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575690.949651:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575690.949657:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575690/real 1788575690] req@ffff94d4049bd880 x1875450488928896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575690.949664:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd880 x1875450488928896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575690.949669:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575690.949670:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575690.949672:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488928896:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575690.949690:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488929024:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575690.949699:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575690.949705:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575690/real 1788575690] req@ffff94d4049bd500 x1875450488929024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575690.949711:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488929024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575690.949715:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575690.949717:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575690.949718:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488929024:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575690.949722:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488929152:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575690.949731:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575690.949736:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575690/real 1788575690] req@ffff94d4049bdc00 x1875450488929152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575690.949742:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488929152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575690.949746:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575690.949748:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575690.949749:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488929152:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575690.949752:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488929280:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575690.949759:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575690.949764:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575690/real 1788575690] req@ffff94d4049be680 x1875450488929280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575690.949769:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488929280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575690.949773:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575690.949775:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575690.949776:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488929280:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575691.461519:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575691.909361:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575692.485678:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575692.933381:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575693.509753:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575693.957380:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575694.533636:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575694.981381:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575695.557294:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575696.005377:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575696.069311:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575696.069326:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575696.069331:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575696.069369:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6844 00000100:00080000:3.0:1788575696.069383:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575696.069392:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575696.069442:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488929408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575696.069473:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575696.069477:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575696.069479:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575696.069483:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6844 00000100:00080000:3.0:1788575696.069486:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575696.069491:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575696.069501:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575696.069517:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488929536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575696.069528:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575696.069534:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575696.069536:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575696.069540:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6844 00000100:00080000:3.0:1788575696.069545:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575696.069550:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575696.069555:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575696.069564:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488929408:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575696.069576:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488929664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575696.069590:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575696.069595:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575696.069596:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575696.069601:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6844 00000100:00080000:3.0:1788575696.069607:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575696.069612:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575696.069619:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575696.069639:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488929792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788575696.069657:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575696.069664:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575696.069669:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575696.069671:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575696.069677:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6844 00000100:00080000:3.0:1788575696.069686:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575696.069691:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575696.069697:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575696.069703:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575696/real 1788575696] req@ffff94d408c55f80 x1875450488929408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575696.069711:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488929920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575696.069744:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488929408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575696.069757:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575696.069762:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575696.069766:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488929408:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575696.069774:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488929536:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575696.069797:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575696.069809:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575696/real 1788575696] req@ffff94d408c54700 x1875450488929536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575696.069817:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488929536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575696.069822:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575696.069824:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575696.069826:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488929536:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575696.069847:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488929664:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575696.069860:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575696.069866:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575696/real 1788575696] req@ffff94d408c56300 x1875450488929664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575696.069876:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488929664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575696.069884:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575696.069887:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575696.069890:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488929664:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575696.069895:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488929792:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575696.069914:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575696.069933:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575696/real 1788575696] req@ffff94d408c55880 x1875450488929792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575696.069955:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488929792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575696.069962:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575696.069965:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575696.069968:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488929792:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575696.069974:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488929920:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575696.069993:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575696.070004:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575696/real 1788575696] req@ffff94d408c55180 x1875450488929920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575696.070015:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488929920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575696.070027:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575696.070030:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575696.070033:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488929920:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575696.581334:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575697.029461:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575697.605293:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575698.053386:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575698.629372:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575699.077434:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575699.653416:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575700.101475:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575700.677357:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575701.125532:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575701.189330:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575701.189371:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575701.189376:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575701.189385:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6849 00000100:00080000:0.0:1788575701.189395:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575701.189402:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575701.189444:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488930048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575701.189469:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575701.189472:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575701.189473:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575701.189477:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6849 00000100:00080000:0.0:1788575701.189480:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575701.189483:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575701.189491:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575701.189502:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488930176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575701.189511:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575701.189514:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575701.189516:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575701.189521:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6849 00000100:00080000:0.0:1788575701.189524:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575701.189527:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575701.189531:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575701.189540:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488930304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575701.189554:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575701.189561:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1788575701.189563:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488930048:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788575701.189564:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575701.189572:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6849 00000100:00080000:0.0:1788575701.189580:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575701.189584:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575701.189589:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575701.189599:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488930432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575701.189606:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575701.189609:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575701.189611:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575701.189614:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6849 00000100:00080000:0.0:1788575701.189617:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575701.189621:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575701.189625:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575701.189634:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488930560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788575701.189669:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575701.189724:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575701/real 1788575701] req@ffff94d408d3ca80 x1875450488930048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575701.189745:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488930048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575701.189758:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575701.189763:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575701.189767:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488930048:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575701.189776:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488930176:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575701.189802:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575701.189815:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575701/real 1788575701] req@ffff94d408d3ea00 x1875450488930176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575701.189826:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488930176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575701.189835:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575701.189838:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575701.189841:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488930176:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575701.189869:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488930304:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575701.189887:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575701.189896:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575701/real 1788575701] req@ffff94d408d3e680 x1875450488930304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575701.189906:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488930304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575701.189923:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575701.189927:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575701.189930:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488930304:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575701.189936:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488930432:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575701.189955:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575701.189965:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575701/real 1788575701] req@ffff94d408d3c000 x1875450488930432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575701.189975:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488930432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575701.189982:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575701.189985:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575701.189988:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488930432:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575701.189993:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488930560:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575701.190009:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575701.190017:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575701/real 1788575701] req@ffff94d408d3d500 x1875450488930560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575701.190027:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488930560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575701.190035:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575701.190038:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575701.190041:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488930560:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575701.701363:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575702.149733:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575702.725373:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575703.173418:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575703.749353:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575704.197412:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575704.773371:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575705.221425:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575705.797319:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575706.245446:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575706.309361:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575706.309372:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575706.309385:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575706.309394:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6854 00000100:00080000:0.0:1788575706.309405:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575706.309413:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575706.309451:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f100 x1875450488930688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575706.309479:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575706.309483:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575706.309485:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575706.309489:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6854 00000100:00080000:0.0:1788575706.309493:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575706.309497:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575706.309506:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575706.309522:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f800 x1875450488930816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575706.309535:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575706.309539:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575706.309542:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575706.309551:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6854 00000100:00100000:2.0:1788575706.309551:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488930688:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788575706.309568:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575706.309575:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575706.309593:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575706.309612:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488930944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575706.309623:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575706.309626:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575706.309628:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575706.309632:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6854 00000100:00080000:0.0:1788575706.309637:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788575706.309642:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788575706.309657:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575706.309666:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575706.309680:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488931072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575706.309696:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788575706.309697:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575706/real 1788575706] req@ffff94d408d3f100 x1875450488930688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575706.309703:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575706.309708:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575706.309720:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6854 00000100:00080000:0.0:1788575706.309727:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575706.309735:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575706.309737:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f100 x1875450488930688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788575706.309744:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788575706.309755:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575706.309762:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575706.309768:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488930688:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575706.309776:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488931200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575706.309806:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488930816:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575706.309837:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575706.309850:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575706/real 1788575706] req@ffff94d408d3f800 x1875450488930816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575706.309897:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f800 x1875450488930816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575706.309907:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575706.309910:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575706.309912:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488930816:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575706.309919:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488930944:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575706.309941:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575706.309953:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575706/real 1788575706] req@ffff94d408d3d500 x1875450488930944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575706.309965:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488930944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575706.309973:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575706.309977:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575706.309980:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488930944:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575706.309987:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488931072:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575706.310008:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575706.310019:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575706/real 1788575706] req@ffff94d408d3c000 x1875450488931072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575706.310050:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488931072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575706.310058:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575706.310061:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575706.310065:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488931072:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575706.310071:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488931200:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575706.310091:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575706.310102:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575706/real 1788575706] req@ffff94d408d3e680 x1875450488931200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575706.310114:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488931200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575706.310122:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575706.310126:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575706.310129:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488931200:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575706.821385:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575707.269373:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575707.845317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575708.293495:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575708.869324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575709.317352:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575709.893318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575710.342388:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575710.917363:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575711.365353:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575711.429305:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575711.429315:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575711.429319:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575711.429329:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6859 00000100:00080000:3.0:1788575711.429348:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575711.429356:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575711.429392:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488931328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575711.429418:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575711.429422:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575711.429424:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575711.429428:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6859 00000100:00080000:3.0:1788575711.429432:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575711.429436:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575711.429446:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575711.429455:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488931328:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575711.429461:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488931456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575711.429473:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575711.429476:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575711.429478:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575711.429481:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6859 00000100:00080000:3.0:1788575711.429485:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575711.429489:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575711.429495:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575711.429507:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488931584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575711.429517:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575711.429522:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575711.429524:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788575711.429528:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575711.429530:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6859 00000100:00080000:3.0:1788575711.429535:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575711.429540:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575711.429546:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575711.429557:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488931712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575711.429566:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788575711.429569:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575711/real 1788575711] req@ffff94d408c54000 x1875450488931328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575711.429571:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575711.429575:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575711.429584:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6859 00000100:00080000:3.0:1788575711.429588:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788575711.429592:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488931328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575711.429596:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575711.429605:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788575711.429614:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575711.429618:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575711.429621:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488931840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575711.429624:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488931328:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575711.429648:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488931456:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575711.429671:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575711.429682:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575711/real 1788575711] req@ffff94d408c57480 x1875450488931456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575711.429694:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488931456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575711.429702:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575711.429704:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575711.429706:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488931456:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575711.429710:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488931584:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575711.429723:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575711.429730:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575711/real 1788575711] req@ffff94d408c55180 x1875450488931584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575711.429737:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488931584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575711.429741:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575711.429743:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575711.429745:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488931584:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575711.429748:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488931712:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575711.429759:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575711.429767:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575711/real 1788575711] req@ffff94d408c55880 x1875450488931712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575711.429777:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488931712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575711.429784:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575711.429787:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575711.429789:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488931712:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575711.429794:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488931840:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575711.429812:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575711.429823:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575711/real 1788575711] req@ffff94d408c56300 x1875450488931840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575711.429832:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488931840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575711.429837:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575711.429839:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575711.429840:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488931840:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575711.941300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575712.389356:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575712.965287:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575713.413366:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575713.989316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575714.437398:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575715.013372:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575715.461415:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575716.037313:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575716.485396:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575716.550237:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575716.550248:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575716.550253:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575716.550263:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6864 00000100:00080000:3.0:1788575716.550273:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575716.550280:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575716.550317:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488931968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575716.550357:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575716.550361:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575716.550363:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575716.550367:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6864 00000100:00080000:3.0:1788575716.550370:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575716.550375:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575716.550383:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575716.550401:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488932096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575716.550408:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575716.550410:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575716.550411:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575716.550414:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6864 00000100:00080000:3.0:1788575716.550416:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575716.550418:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575716.550421:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575716.550428:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488932224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575716.550433:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575716.550435:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575716.550436:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575716.550438:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6864 00000100:00080000:3.0:1788575716.550439:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575716.550441:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575716.550444:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575716.550448:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488932352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575716.550453:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575716.550455:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575716.550455:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575716.550457:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6864 00000100:00080000:3.0:1788575716.550458:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575716.550460:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575716.550462:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575716.550467:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488932480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575716.550500:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488931968:192.168.202.125@tcp:250:kworker.0 00000800:00000100:2.0:1788575716.550565:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575716.550596:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575716/real 1788575716] req@ffff94d408c54700 x1875450488931968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575716.550610:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488931968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575716.550618:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575716.550621:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575716.550624:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488931968:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575716.550629:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488932096:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575716.550652:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575716.550669:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575716/real 1788575716] req@ffff94d408c55f80 x1875450488932096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575716.550685:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488932096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575716.550693:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575716.550696:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575716.550700:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488932096:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575716.550707:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488932224:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575716.550727:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575716.550738:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575716/real 1788575716] req@ffff94d408c55c00 x1875450488932224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575716.550749:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488932224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575716.550760:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575716.550762:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575716.550764:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488932224:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575716.550770:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488932352:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575716.550802:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575716.550816:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575716/real 1788575716] req@ffff94d408c54e00 x1875450488932352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575716.550835:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488932352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575716.550844:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575716.550847:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575716.550851:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488932352:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575716.550859:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488932480:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575716.550887:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575716.550903:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575716/real 1788575716] req@ffff94d408c56a00 x1875450488932480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575716.550918:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488932480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575716.550926:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575716.550930:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575716.550934:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488932480:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575717.061340:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575717.509464:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575718.085330:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575718.533412:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575719.110264:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575719.557372:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575720.133317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575720.582225:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575721.157314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575721.606327:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575721.669280:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575721.669288:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575721.669292:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575721.669309:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6869 00000100:00080000:0.0:1788575721.669323:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575721.669330:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575721.669361:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488932608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575721.669385:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575721.669388:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575721.669389:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575721.669391:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6869 00000100:00080000:0.0:1788575721.669394:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575721.669396:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575721.669403:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575721.669415:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488932736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575721.669425:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788575721.669428:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488932608:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788575721.669430:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575721.669434:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575721.669439:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6869 00000100:00080000:0.0:1788575721.669443:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575721.669447:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575721.669451:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575721.669461:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488932864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575721.669468:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575721.669470:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575721.669471:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575721.669474:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6869 00000100:00080000:0.0:1788575721.669476:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575721.669479:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575721.669490:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788575721.669494:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575721.669499:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488932992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575721.669506:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575721.669508:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575721.669509:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575721.669511:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6869 00000100:00080000:0.0:1788575721.669514:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575721.669516:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575721.669519:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575721.669525:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488933120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575721.669533:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575721/real 1788575721] req@ffff94d408d3ea00 x1875450488932608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575721.669549:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488932608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575721.669556:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575721.669559:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575721.669561:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488932608:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575721.669578:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488932736:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575721.669594:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575721.669602:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575721/real 1788575721] req@ffff94d408d3ca80 x1875450488932736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575721.669615:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488932736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575721.669620:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575721.669622:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575721.669624:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488932736:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575721.669627:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488932864:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575721.669636:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575721.669642:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575721/real 1788575721] req@ffff94d408d3d180 x1875450488932864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575721.669649:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488932864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575721.669653:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575721.669655:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575721.669657:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488932864:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575721.669669:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488932992:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575721.669692:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575721.669703:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575721/real 1788575721] req@ffff94d408d3df80 x1875450488932992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575721.669714:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488932992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575721.669719:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575721.669721:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575721.669723:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488932992:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575721.669730:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488933120:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575721.669743:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575721.669749:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575721/real 1788575721] req@ffff94d408d3ce00 x1875450488933120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575721.669756:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488933120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575721.669760:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575721.669762:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575721.669764:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488933120:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575722.181302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575722.629396:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575723.205343:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575723.653418:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575724.229329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575724.677336:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575725.253297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575725.701372:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575726.277320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575726.725376:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575726.789333:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575726.789343:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575726.789347:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575726.789357:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6875 00000100:00080000:3.0:1788575726.789369:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575726.789375:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575726.789405:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488933248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575726.789428:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575726.789430:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575726.789432:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575726.789435:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6875 00000100:00080000:3.0:1788575726.789437:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575726.789439:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575726.789445:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575726.789453:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488933376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575726.789460:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575726.789462:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575726.789463:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575726.789466:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6875 00000100:00080000:3.0:1788575726.789467:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575726.789469:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575726.789473:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575726.789476:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488933248:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575726.789482:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488933504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575726.789489:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575726.789491:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575726.789492:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575726.789494:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6875 00000100:00080000:3.0:1788575726.789496:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575726.789498:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575726.789500:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575726.789508:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488933632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575726.789513:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575726.789516:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575726.789517:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575726.789518:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6875 00000100:00080000:3.0:1788575726.789526:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575726.789529:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575726.789531:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788575726.789536:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575726.789539:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488933760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575726.789570:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575726/real 1788575726] req@ffff94d408c54380 x1875450488933248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575726.789583:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488933248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575726.789590:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575726.789593:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575726.789595:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488933248:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575726.789599:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488933376:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575726.789614:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575726.789620:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575726/real 1788575726] req@ffff94d408c57800 x1875450488933376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575726.789627:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488933376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575726.789640:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575726.789642:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575726.789644:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488933376:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575726.789668:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488933504:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575726.789686:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575726.789694:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575726/real 1788575726] req@ffff94d408c57b80 x1875450488933504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575726.789701:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488933504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575726.789706:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575726.789708:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575726.789710:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488933504:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575726.789714:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488933632:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575726.789724:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575726.789729:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575726/real 1788575726] req@ffff94d408c56d80 x1875450488933632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575726.789736:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488933632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575726.789740:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575726.789742:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575726.789743:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488933632:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575726.789746:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488933760:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575726.789754:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575726.789759:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575726/real 1788575726] req@ffff94d408c56680 x1875450488933760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575726.789766:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488933760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575726.789770:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575726.789771:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575726.789773:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488933760:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575727.301320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575727.749354:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575728.325343:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575728.773410:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575729.349304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575729.797385:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575730.373324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575730.821594:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575731.397322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575731.845399:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575731.909317:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575731.909328:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575731.909332:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575731.909342:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6880 00000100:00080000:3.0:1788575731.909354:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575731.909361:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575731.909389:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488933888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575731.909416:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575731.909419:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575731.909421:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575731.909425:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6880 00000100:00080000:3.0:1788575731.909428:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575731.909432:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575731.909440:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575731.909453:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488934016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575731.909465:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788575731.909467:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488933888:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575731.909470:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575731.909475:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575731.909480:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6880 00000100:00080000:3.0:1788575731.909484:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575731.909488:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575731.909493:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575731.909506:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488934144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575731.909529:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575731.909532:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575731.909536:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788575731.909537:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575731.909544:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6880 00000100:00080000:3.0:1788575731.909550:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575731.909555:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575731.909561:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575731.909574:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488934272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575731.909578:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575731/real 1788575731] req@ffff94d408c57100 x1875450488933888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575731.909604:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575731.909609:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575731.909612:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788575731.909618:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488933888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575731.909618:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6880 00000100:00080000:3.0:1788575731.909623:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575731.909631:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00080000:2.0:1788575731.909635:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788575731.909638:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788575731.909642:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575731.909647:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488933888:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575731.909652:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488934400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575731.909683:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488934016:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575731.909708:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575731.909720:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575731/real 1788575731] req@ffff94d408c54a80 x1875450488934016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575731.909731:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488934016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575731.909755:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575731.909758:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575731.909762:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488934016:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575731.909767:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488934144:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575731.909786:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575731.909796:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575731/real 1788575731] req@ffff94d408c56680 x1875450488934144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575731.909807:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488934144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575731.909814:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575731.909817:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575731.909821:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488934144:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575731.909826:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488934272:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575731.909843:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575731.909852:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575731/real 1788575731] req@ffff94d408c56d80 x1875450488934272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575731.909862:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488934272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575731.909869:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575731.909872:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575731.909876:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488934272:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575731.909881:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488934400:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575731.909898:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575731.909908:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575731/real 1788575731] req@ffff94d408c57b80 x1875450488934400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575731.909919:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488934400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575731.909926:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575731.909928:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575731.909932:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488934400:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575732.421300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575732.869413:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575733.445344:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575733.893525:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575734.469325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575734.917382:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575735.493364:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575735.941533:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575736.517375:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575736.965446:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575737.029495:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575737.029510:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575737.029515:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575737.029529:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6885 00000100:00080000:0.0:1788575737.029545:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575737.029555:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575737.029605:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488934528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575737.029638:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575737.029643:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575737.029645:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575737.029649:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6885 00000100:00080000:0.0:1788575737.029653:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575737.029657:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575737.029668:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575737.029685:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488934656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575737.029698:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788575737.029698:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488934528:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788575737.029708:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575737.029711:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575737.029715:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6885 00000100:00080000:0.0:1788575737.029718:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575737.029722:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575737.029727:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575737.029740:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ed80 x1875450488934784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575737.029749:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575737.029753:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575737.029755:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575737.029759:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6885 00000100:00080000:0.0:1788575737.029763:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575737.029767:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575737.029778:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788575737.029785:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575737.029793:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488934912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575737.029803:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575737.029807:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575737.029808:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575737.029813:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6885 00000100:00080000:0.0:1788575737.029816:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575737.029820:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575737.029825:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575737.029835:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488935040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575737.029890:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575737/real 1788575737] req@ffff94d408d3f480 x1875450488934528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575737.029909:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488934528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575737.029921:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575737.029925:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575737.029929:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488934528:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575737.029956:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488934656:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575737.029977:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575737.029987:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575737/real 1788575737] req@ffff94d408d3c700 x1875450488934656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575737.029994:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488934656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575737.030000:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575737.030002:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575737.030004:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488934656:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575737.030015:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488934784:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575737.030033:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575737.030049:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575737/real 1788575737] req@ffff94d408d3ed80 x1875450488934784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575737.030062:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ed80 x1875450488934784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575737.030069:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575737.030071:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575737.030073:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488934784:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575737.030078:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488934912:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575737.030093:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575737.030099:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575737/real 1788575737] req@ffff94d408d3c380 x1875450488934912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575737.030106:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488934912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575737.030112:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575737.030114:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575737.030116:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488934912:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575737.030119:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488935040:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575737.030127:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575737.030132:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575737/real 1788575737] req@ffff94d408d3d880 x1875450488935040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575737.030138:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488935040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575737.030143:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575737.030145:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575737.030146:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488935040:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575737.541338:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575737.989504:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575738.565372:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575739.013518:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575739.589366:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575740.037456:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575740.613380:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575741.061456:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575741.638189:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575742.085418:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575742.149315:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575742.149325:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575742.149328:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575742.149338:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6890 00000100:00080000:0.0:1788575742.149348:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575742.149356:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575742.149391:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488935168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575742.149420:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575742.149424:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575742.149426:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575742.149431:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6890 00000100:00080000:0.0:1788575742.149435:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575742.149448:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575742.149458:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575742.149473:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488935296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575742.149483:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575742.149486:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575742.149488:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575742.149492:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6890 00000100:00080000:0.0:1788575742.149495:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575742.149499:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575742.149504:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575742.149512:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488935168:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575742.149522:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488935424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575742.149536:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575742.149539:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575742.149541:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575742.149546:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6890 00000100:00080000:0.0:1788575742.149550:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575742.149554:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575742.149561:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575742.149578:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488935552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575742.149588:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575742.149592:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575742.149594:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575742.149598:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6890 00000100:00080000:0.0:1788575742.149602:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575742.149606:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575742.149611:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788575742.149613:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575742.149629:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ed80 x1875450488935680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575742.149671:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575742/real 1788575742] req@ffff94d408d3dc00 x1875450488935168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575742.149694:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488935168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575742.149707:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575742.149712:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575742.149716:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488935168:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575742.149726:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488935296:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575742.149751:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575742.149763:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575742/real 1788575742] req@ffff94d408d3e300 x1875450488935296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575742.149775:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488935296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575742.149783:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575742.149787:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575742.149804:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488935296:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575742.149835:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488935424:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575742.149858:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575742.149870:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575742/real 1788575742] req@ffff94d408d3d880 x1875450488935424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575742.149882:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488935424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575742.149891:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575742.149894:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575742.149898:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488935424:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575742.149904:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488935552:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575742.149924:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575742.149935:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575742/real 1788575742] req@ffff94d408d3c380 x1875450488935552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575742.149947:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488935552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575742.149954:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575742.149958:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575742.149960:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488935552:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575742.149967:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488935680:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575742.149986:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575742.149997:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575742/real 1788575742] req@ffff94d408d3ed80 x1875450488935680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575742.150008:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ed80 x1875450488935680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575742.150016:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575742.150019:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575742.150022:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488935680:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575742.661340:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575743.109397:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575743.685347:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575744.133381:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575744.709333:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575745.157429:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575745.733351:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575746.181366:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575746.757277:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575747.205372:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575747.269335:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575747.269345:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575747.269349:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575747.269359:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6895 00000100:00080000:0.0:1788575747.269370:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575747.269377:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575747.269414:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488935808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575747.269448:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575747.269451:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575747.269453:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575747.269456:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6895 00000100:00080000:0.0:1788575747.269459:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575747.269463:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575747.269472:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575747.269483:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488935936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575747.269487:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488935808:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788575747.269502:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575747.269505:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575747.269507:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575747.269511:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6895 00000100:00080000:0.0:1788575747.269515:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575747.269518:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575747.269522:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575747.269532:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488936064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575747.269541:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575747.269544:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575747.269546:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575747.269550:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6895 00000100:00080000:0.0:1788575747.269554:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575747.269558:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575747.269562:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575747.269571:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488936192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575747.269581:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575747.269584:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575747.269585:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575747.269588:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6895 00000100:00080000:0.0:1788575747.269591:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575747.269597:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575747.269597:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488935936:192.168.202.125@tcp:38:kworker.0 00000008:00100000:0.0:1788575747.269605:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575747.269618:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488936320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575747.269641:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488936064:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575747.269660:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488936192:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575747.269676:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488936320:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575747.781306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575748.229382:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575748.805310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575749.253414:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575749.829291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575750.277368:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788575750.341433:0:2392:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:1.0:1788575750.341474:0:2392:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788575750.341488:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575750.341522:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575750.341533:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575750.341543:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575750.341552:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:2.0:1788575750.341564:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575747/real 1788575750] req@ffff94d408d3f480 x1875450488935936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575750.341589:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488935936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575750.341600:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575750.341606:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575750.341609:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488935936:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575750.341617:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575747/real 1788575750] req@ffff94d408d3ce00 x1875450488936064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575750.341642:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488936064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575750.341649:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575750.341652:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575750.341655:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488936064:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575750.341660:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575747/real 1788575750] req@ffff94d408d3df80 x1875450488936192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575750.341671:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488936192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575750.341677:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575750.341680:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575750.341693:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488936192:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575750.341699:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575747/real 1788575750] req@ffff94d408d3c700 x1875450488935808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575750.341710:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488935808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575750.341716:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575750.341719:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575750.341722:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488935808:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575750.341745:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575747/real 1788575750] req@ffff94d408d3d180 x1875450488936320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575750.341756:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488936320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575750.341762:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575750.341764:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575750.341767:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488936320:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575750.853480:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575751.301380:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575751.877288:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575752.325338:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575752.389410:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575752.389419:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575752.389423:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575752.389434:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6900 00000100:00080000:1.0:1788575752.389443:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575752.389449:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575752.389480:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488936448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575752.389510:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575752.389514:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575752.389517:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575752.389522:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6900 00000100:00080000:1.0:1788575752.389526:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575752.389530:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575752.389541:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575752.389554:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488936576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575752.389565:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575752.389568:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575752.389570:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575752.389574:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6900 00000100:00080000:1.0:1788575752.389577:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575752.389580:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575752.389597:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575752.389613:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488936704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575752.389623:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575752.389626:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575752.389628:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575752.389631:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6900 00000100:00080000:1.0:1788575752.389639:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575752.389645:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575752.389650:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575752.389662:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488936832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575752.389672:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575752.389675:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575752.389677:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575752.389681:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6900 00000100:00080000:1.0:1788575752.389685:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575752.389689:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575752.389695:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575752.389706:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488936960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575752.390230:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488936448:192.168.202.125@tcp:250:kworker.0 00000800:00000100:3.0:1788575752.390537:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575752.390591:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575752/real 1788575752] req@ffff94d4049bf100 x1875450488936448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575752.390614:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488936448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575752.390633:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575752.390638:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575752.390642:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488936448:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575752.390652:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488936576:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575752.390682:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575752.390696:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575752/real 1788575752] req@ffff94d4049be300 x1875450488936576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575752.390708:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488936576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575752.390716:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575752.390719:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575752.390722:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488936576:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575752.390729:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488936704:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575752.390749:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575752.390760:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575752/real 1788575752] req@ffff94d4049bf800 x1875450488936704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575752.390772:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488936704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575752.390779:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575752.390782:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575752.390785:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488936704:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575752.390792:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488936832:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575752.390812:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575752.390823:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575752/real 1788575752] req@ffff94d4049bed80 x1875450488936832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575752.390834:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488936832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575752.390842:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575752.390845:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575752.390848:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488936832:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575752.390854:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488936960:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575752.390872:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575752.390882:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575752/real 1788575752] req@ffff94d4049bc700 x1875450488936960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575752.390893:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488936960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575752.390901:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575752.390904:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575752.390907:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488936960:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575752.901321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575753.349392:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575753.925278:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575754.373367:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575754.949380:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575755.397763:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575755.973365:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575756.421438:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575756.998613:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575757.445382:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575757.509293:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575757.509304:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575757.509308:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575757.509318:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6905 00000100:00080000:1.0:1788575757.509330:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575757.509348:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575757.509378:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488937088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575757.509400:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575757.509403:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575757.509404:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575757.509406:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6905 00000100:00080000:1.0:1788575757.509409:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575757.509411:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575757.509418:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575757.509425:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488937216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575757.509431:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575757.509435:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575757.509437:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575757.509439:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6905 00000100:00080000:1.0:1788575757.509441:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575757.509444:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575757.509448:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575757.509452:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488937088:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575757.509461:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488937344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575757.509472:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575757.509476:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575757.509477:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575757.509481:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6905 00000100:00080000:1.0:1788575757.509485:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575757.509488:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575757.509494:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575757.509505:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488937472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575757.509515:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788575757.509516:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788575757.509522:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575757.509524:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575757.509529:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6905 00000100:00080000:1.0:1788575757.509533:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575757.509538:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575757.509543:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575757.509555:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575757/real 1788575757] req@ffff94d4049bc000 x1875450488937088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575757.509556:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488937600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575757.509595:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488937088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575757.509605:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575757.509609:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575757.509613:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488937088:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575757.509619:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488937216:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575757.509640:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575757.509663:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575757/real 1788575757] req@ffff94d4049bfb80 x1875450488937216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575757.509675:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488937216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575757.509683:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575757.509685:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575757.509689:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488937216:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575757.509710:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488937344:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575757.509729:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575757.509740:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575757/real 1788575757] req@ffff94d4049bd180 x1875450488937344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575757.509750:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488937344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575757.509757:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575757.509759:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575757.509762:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488937344:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575757.509768:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488937472:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575757.509784:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575757.509808:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575757/real 1788575757] req@ffff94d4049bce00 x1875450488937472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575757.509820:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488937472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575757.509827:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575757.509831:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575757.509834:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488937472:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575757.509839:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488937600:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575757.509856:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575757.509866:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575757/real 1788575757] req@ffff94d4049bea00 x1875450488937600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575757.509876:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488937600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575757.509883:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575757.509886:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575757.509888:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488937600:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575758.021945:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575758.469396:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575759.047925:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575759.493353:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575760.069465:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575760.517458:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575761.093764:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575761.541448:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575762.117350:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575762.565418:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575762.629339:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575762.629352:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575762.629357:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575762.629369:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6910 00000100:00080000:0.0:1788575762.629382:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575762.629391:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575762.629446:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488937728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575762.629475:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575762.629479:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575762.629481:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575762.629485:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6910 00000100:00080000:0.0:1788575762.629488:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575762.629492:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575762.629502:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575762.629514:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488937856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575762.629526:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575762.629530:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575762.629533:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788575762.629535:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488937728:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788575762.629538:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6910 00000100:00080000:0.0:1788575762.629543:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575762.629547:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575762.629551:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575762.629565:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488937984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575762.629582:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575762.629586:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575762.629588:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575762.629592:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6910 00000100:00080000:0.0:1788575762.629595:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575762.629599:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575762.629603:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575762.629614:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488938112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575762.629624:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788575762.629635:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575762.629638:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575762.629640:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575762.629645:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6910 00000100:00080000:0.0:1788575762.629650:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575762.629655:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575762.629660:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575762.629676:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575762/real 1788575762] req@ffff94d408d3ca80 x1875450488937728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575762.629680:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488938240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575762.629710:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488937728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575762.629721:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575762.629726:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575762.629730:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488937728:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575762.629756:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488937856:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575762.629779:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575762.629792:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575762/real 1788575762] req@ffff94d408d3ea00 x1875450488937856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575762.629810:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488937856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575762.629818:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575762.629821:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575762.629825:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488937856:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575762.629830:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488937984:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575762.629850:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575762.629861:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575762/real 1788575762] req@ffff94d408d3e680 x1875450488937984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575762.629873:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488937984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575762.629881:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575762.629884:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575762.629887:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488937984:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575762.629892:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488938112:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575762.629908:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575762.629916:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575762/real 1788575762] req@ffff94d408d3c000 x1875450488938112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575762.629928:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488938112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575762.629940:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575762.629943:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575762.629946:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488938112:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575762.629952:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488938240:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575762.629970:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575762.629980:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575762/real 1788575762] req@ffff94d408d3d500 x1875450488938240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575762.629990:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488938240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575762.629997:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575762.630000:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575762.630003:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488938240:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575763.141489:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575763.589417:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575764.165346:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575764.613413:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575765.189314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575765.637387:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575766.213312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575766.661416:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575767.238013:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575767.685369:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575767.749316:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575767.749326:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575767.749330:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575767.749341:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6916 00000100:00080000:1.0:1788575767.749352:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575767.749359:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575767.749399:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488938368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575767.749422:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575767.749425:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575767.749426:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575767.749429:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6916 00000100:00080000:1.0:1788575767.749431:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575767.749434:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575767.749442:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575767.749451:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488938496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575767.749458:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575767.749461:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788575767.749461:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488938368:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788575767.749465:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575767.749471:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6916 00000100:00080000:1.0:1788575767.749474:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575767.749476:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575767.749479:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575767.749490:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488938624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575767.749495:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575767.749497:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575767.749498:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575767.749501:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6916 00000100:00080000:1.0:1788575767.749503:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575767.749506:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575767.749510:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575767.749516:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488938752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575767.749521:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575767.749523:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575767.749524:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575767.749526:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6916 00000100:00080000:1.0:1788575767.749528:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575767.749532:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:3.0:1788575767.749533:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:1.0:1788575767.749535:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575767.749543:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488938880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575767.749571:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575767/real 1788575767] req@ffff94d4049bf480 x1875450488938368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575767.749593:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488938368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575767.749604:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575767.749608:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575767.749612:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488938368:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575767.749637:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488938496:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575767.749658:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575767.749669:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575767/real 1788575767] req@ffff94d4049bc380 x1875450488938496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575767.749680:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc380 x1875450488938496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575767.749687:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575767.749690:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575767.749693:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488938496:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575767.749699:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488938624:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575767.749714:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575767.749722:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575767/real 1788575767] req@ffff94d4049bea00 x1875450488938624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575767.749741:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488938624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575767.749749:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575767.749751:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575767.749754:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488938624:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575767.749760:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488938752:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575767.749777:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575767.749786:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575767/real 1788575767] req@ffff94d4049bce00 x1875450488938752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575767.749804:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488938752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575767.749814:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575767.749817:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575767.749820:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488938752:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575767.749825:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488938880:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575767.749852:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575767.749861:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575767/real 1788575767] req@ffff94d4049bd180 x1875450488938880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575767.749871:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488938880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575767.749878:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575767.749882:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575767.749884:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488938880:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575768.261321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575768.709438:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575769.285327:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575769.733338:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575770.309336:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575770.757436:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575771.333359:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575771.781376:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575772.357283:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575772.805430:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575772.869362:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575772.869374:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575772.869380:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575772.869392:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6921 00000100:00080000:1.0:1788575772.869406:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575772.869415:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575772.869465:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488939008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575772.869498:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575772.869502:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575772.869504:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575772.869508:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6921 00000100:00080000:1.0:1788575772.869512:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575772.869516:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575772.869528:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575772.869542:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488939136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575772.869554:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575772.869558:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575772.869561:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575772.869569:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6921 00000100:00100000:3.0:1788575772.869569:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488939008:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788575772.869598:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575772.869609:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575772.869616:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575772.869632:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488939264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575772.869644:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575772.869648:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575772.869649:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575772.869654:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6921 00000100:00080000:1.0:1788575772.869658:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788575772.869664:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788575772.869673:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575772.869684:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575772.869701:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575772/real 1788575772] req@ffff94d4049bfb80 x1875450488939008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575772.869707:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488939392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575772.869759:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575772.869765:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575772.869767:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788575772.869769:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488939008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575772.869784:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6921 00000100:00080000:3.0:1788575772.869788:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575772.869791:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575772.869795:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:1.0:1788575772.869803:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575772.869804:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488939008:192.168.202.125@tcp:250:kworker.0 00000008:00100000:1.0:1788575772.869820:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575772.869836:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488939136:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575772.869838:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488939520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575772.869857:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575772.869868:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575772/real 1788575772] req@ffff94d4049bc000 x1875450488939136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575772.869877:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488939136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575772.869885:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575772.869888:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575772.869890:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488939136:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575772.869895:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488939264:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575772.869906:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575772.869911:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575772/real 1788575772] req@ffff94d4049bc700 x1875450488939264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575772.869917:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488939264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575772.869926:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575772.869928:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575772.869930:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488939264:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575772.869933:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488939392:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575772.869942:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575772.869947:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575772/real 1788575772] req@ffff94d4049bed80 x1875450488939392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575772.869953:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488939392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575772.869958:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575772.869960:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575772.869961:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488939392:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575772.869970:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488939520:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575772.869979:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575772.869984:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575772/real 1788575772] req@ffff94d4049bf800 x1875450488939520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575772.869990:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488939520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575772.869994:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575772.869996:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575772.869998:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488939520:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575773.381304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575773.829380:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575774.405376:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575774.853438:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575775.429365:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575775.877461:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575776.453318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575776.901327:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575777.477295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575777.925370:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575777.989314:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575777.989324:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575777.989329:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575777.989363:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6926 00000100:00080000:1.0:1788575777.989375:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575777.989382:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575777.989421:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488939648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575777.989450:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575777.989454:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575777.989456:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575777.989460:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6926 00000100:00080000:1.0:1788575777.989463:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575777.989467:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575777.989476:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575777.989489:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488939776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575777.989499:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575777.989502:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575777.989504:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575777.989509:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6926 00000100:00100000:3.0:1788575777.989510:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488939648:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788575777.989513:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575777.989520:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575777.989525:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575777.989536:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488939904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575777.989550:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575777.989553:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575777.989555:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575777.989559:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6926 00000100:00080000:1.0:1788575777.989563:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575777.989566:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575777.989571:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575777.989585:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488940032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575777.989586:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788575777.989596:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575777.989600:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575777.989602:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575777.989606:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6926 00000100:00080000:1.0:1788575777.989609:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575777.989613:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575777.989618:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575777.989625:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575777/real 1788575777] req@ffff94d4049be300 x1875450488939648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575777.989630:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488940160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575777.989649:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488939648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575777.989666:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575777.989670:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575777.989673:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488939648:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575777.989697:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488939776:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575777.989718:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575777.989729:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575777/real 1788575777] req@ffff94d4049bf100 x1875450488939776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575777.989740:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488939776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575777.989748:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575777.989751:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575777.989754:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488939776:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575777.989759:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488939904:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575777.989777:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575777.989787:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575777/real 1788575777] req@ffff94d4049be680 x1875450488939904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575777.989798:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488939904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575777.989806:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575777.989809:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575777.989812:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488939904:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575777.989817:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488940032:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575777.989835:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575777.989844:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575777/real 1788575777] req@ffff94d4049bdc00 x1875450488940032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575777.989855:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488940032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575777.989861:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575777.989864:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575777.989867:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488940032:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575777.989872:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488940160:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575777.989889:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575777.989899:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575777/real 1788575777] req@ffff94d4049bd500 x1875450488940160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575777.989928:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488940160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575777.989935:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575777.989937:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575777.989940:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488940160:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575778.501298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575778.949379:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575779.525490:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575779.973362:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575780.549300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575780.997447:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575781.573419:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575782.022354:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575782.597356:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575783.045357:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575783.109379:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575783.109388:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575783.109392:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575783.109401:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6931 00000100:00080000:2.0:1788575783.109412:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575783.109418:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575783.109452:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488940288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575783.109484:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575783.109492:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575783.109494:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575783.109498:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6931 00000100:00080000:2.0:1788575783.109501:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575783.109505:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575783.109514:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575783.109524:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488940288:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575783.109530:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488940416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575783.109541:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575783.109544:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575783.109546:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575783.109549:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6931 00000100:00080000:2.0:1788575783.109553:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575783.109556:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575783.109560:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575783.109570:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488940544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575783.109576:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788575783.109584:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575783.109587:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575783.109589:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575783.109594:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6931 00000100:00080000:2.0:1788575783.109598:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575783.109602:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575783.109607:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575783.109608:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575783/real 1788575783] req@ffff94d408c60a80 x1875450488940288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575783.109618:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488940672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575783.109633:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575783.109636:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488940288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575783.109638:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575783.109642:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575783.109651:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6931 00000100:00080000:3.0:1788575783.109656:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575783.109657:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575783.109665:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:2.0:1788575783.109666:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575783.109675:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575783.109675:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488940288:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575783.109690:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488940800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575783.109705:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488940416:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575783.109731:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575783.109743:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575783/real 1788575783] req@ffff94d408c63b80 x1875450488940416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575783.109755:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488940416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575783.109762:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575783.109765:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575783.109768:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488940416:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575783.109774:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488940544:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575783.109791:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575783.109801:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575783/real 1788575783] req@ffff94d408c62a00 x1875450488940544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575783.109811:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488940544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575783.109818:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575783.109822:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575783.109824:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488940544:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575783.109830:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488940672:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575783.109848:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575783.109858:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575783/real 1788575783] req@ffff94d408c61f80 x1875450488940672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575783.109869:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488940672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575783.109876:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575783.109879:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575783.109882:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488940672:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575783.109888:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488940800:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575783.109906:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575783.109916:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575783/real 1788575783] req@ffff94d408c63480 x1875450488940800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575783.109926:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488940800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575783.109934:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575783.109936:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575783.109939:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488940800:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575783.621352:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575784.069362:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575784.645319:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575785.093369:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575785.669371:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575786.117423:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575786.693308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575787.141403:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575787.717337:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575788.165421:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575788.229308:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575788.229320:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575788.229324:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575788.229334:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6936 00000100:00080000:1.0:1788575788.229346:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575788.229355:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575788.229393:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd880 x1875450488940928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575788.229425:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575788.229429:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575788.229439:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575788.229445:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6936 00000100:00080000:1.0:1788575788.229448:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575788.229452:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575788.229458:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488940928:192.168.202.125@tcp:250:kworker.0 00000008:00100000:1.0:1788575788.229462:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575788.229479:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488941056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575788.229491:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575788.229495:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575788.229497:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575788.229501:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6936 00000100:00080000:1.0:1788575788.229506:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575788.229511:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575788.229516:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575788.229532:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488941184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788575788.229532:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788575788.229542:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575788.229545:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575788.229547:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575788.229549:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6936 00000100:00080000:1.0:1788575788.229553:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575788.229556:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575788.229559:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575788.229568:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488941312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575788.229576:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788575788.229576:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575788/real 1788575788] req@ffff94d4049bd880 x1875450488940928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575788.229582:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575788.229583:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575788.229586:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6936 00000100:00080000:1.0:1788575788.229588:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575788.229590:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575788.229593:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575788.229603:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488941440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575788.229615:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd880 x1875450488940928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575788.229626:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575788.229631:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575788.229635:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488940928:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575788.229659:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488941056:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575788.229685:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575788.229700:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575788/real 1788575788] req@ffff94d4049bca80 x1875450488941056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575788.229713:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488941056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575788.229723:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575788.229728:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575788.229731:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488941056:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575788.229738:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488941184:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575788.229760:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575788.229775:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575788/real 1788575788] req@ffff94d4049bd500 x1875450488941184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575788.229787:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488941184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575788.229795:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575788.229800:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575788.229804:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488941184:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575788.229812:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488941312:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575788.229835:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575788.229847:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575788/real 1788575788] req@ffff94d4049bdc00 x1875450488941312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575788.229859:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488941312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575788.229866:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575788.229869:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575788.229873:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488941312:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575788.229879:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488941440:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575788.229897:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575788.229908:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575788/real 1788575788] req@ffff94d4049be680 x1875450488941440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575788.229918:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488941440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575788.229925:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575788.229928:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575788.229931:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488941440:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575788.741276:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575789.189388:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575789.765330:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575790.213357:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575790.789297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575791.237366:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575791.813324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575792.261352:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575792.837329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575793.285323:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575793.349324:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575793.349334:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575793.349338:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575793.349348:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6941 00000100:00080000:3.0:1788575793.349358:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575793.349365:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575793.349410:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488941568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575793.349436:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575793.349439:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575793.349442:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575793.349446:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6941 00000100:00080000:3.0:1788575793.349450:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575793.349454:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575793.349464:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575793.349477:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488941696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575793.349482:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488941568:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575793.349493:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575793.349497:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575793.349499:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575793.349504:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6941 00000100:00080000:3.0:1788575793.349508:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575793.349511:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575793.349517:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575793.349530:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488941824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575793.349538:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575793.349540:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575793.349541:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575793.349544:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6941 00000800:00000100:2.0:1788575793.349548:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575793.349550:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575793.349555:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575793.349560:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575793.349572:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488941952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575793.349581:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575793.349584:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575793.349586:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788575793.349591:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575793/real 1788575793] req@ffff94d408c57800 x1875450488941568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575793.349592:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6941 00000100:00080000:3.0:1788575793.349598:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575793.349606:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575793.349612:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575793.349618:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488941568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575793.349623:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488942080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575793.349649:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575793.349654:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575793.349659:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488941568:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575793.349680:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488941696:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575793.349702:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575793.349714:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575793/real 1788575793] req@ffff94d408c54380 x1875450488941696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575793.349726:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488941696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575793.349733:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575793.349736:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575793.349740:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488941696:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575793.349745:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488941824:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575793.349763:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575793.349773:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575793/real 1788575793] req@ffff94d408c56a00 x1875450488941824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575793.349785:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488941824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575793.349792:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575793.349795:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575793.349798:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488941824:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575793.349804:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488941952:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575793.349822:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575793.349832:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575793/real 1788575793] req@ffff94d408c54e00 x1875450488941952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575793.349861:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488941952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575793.349868:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575793.349871:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575793.349874:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488941952:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575793.349880:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488942080:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575793.349897:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575793.349907:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575793/real 1788575793] req@ffff94d408c55c00 x1875450488942080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575793.349918:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488942080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575793.349925:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575793.349928:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575793.349931:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488942080:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575793.861360:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575794.309407:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575794.885286:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575795.333424:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575795.909290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575796.357373:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575796.933328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575797.381368:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575797.957291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575798.405412:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575798.469353:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575798.469363:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575798.469367:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575798.469376:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6946 00000100:00080000:1.0:1788575798.469386:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575798.469398:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575798.469441:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488942208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575798.469466:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575798.469473:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575798.469474:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575798.469478:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6946 00000100:00080000:1.0:1788575798.469482:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575798.469486:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575798.469495:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575798.469508:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488942336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575798.469520:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488942208:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788575798.469521:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575798.469529:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575798.469531:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575798.469534:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6946 00000100:00080000:1.0:1788575798.469537:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575798.469540:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575798.469544:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575798.469553:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488942464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575798.469559:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575798.469561:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575798.469562:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575798.469565:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6946 00000100:00080000:1.0:1788575798.469568:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575798.469570:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575798.469573:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575798.469580:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488942592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575798.469585:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575798.469587:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575798.469588:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575798.469590:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6946 00000100:00080000:1.0:1788575798.469591:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575798.469593:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575798.469595:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788575798.469599:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575798.469605:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488942720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575798.469640:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575798/real 1788575798] req@ffff94d4049bf100 x1875450488942208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575798.469659:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488942208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575798.469670:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575798.469674:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575798.469678:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488942208:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575798.469715:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488942336:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575798.469736:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575798.469747:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575798/real 1788575798] req@ffff94d4049be300 x1875450488942336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575798.469759:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488942336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575798.469781:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575798.469784:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575798.469788:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488942336:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575798.469794:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488942464:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575798.469814:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575798.469825:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575798/real 1788575798] req@ffff94d4049bf800 x1875450488942464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575798.469837:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488942464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575798.469851:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575798.469854:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575798.469857:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488942464:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575798.469863:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488942592:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575798.469889:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575798.469901:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575798/real 1788575798] req@ffff94d4049bed80 x1875450488942592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575798.469912:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488942592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575798.469919:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575798.469922:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575798.469925:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488942592:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575798.469931:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488942720:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575798.469949:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575798.469959:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575798/real 1788575798] req@ffff94d4049bc700 x1875450488942720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575798.469970:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488942720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575798.469987:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575798.469990:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575798.469994:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488942720:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575798.981309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575799.429392:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575800.005273:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575800.453365:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575801.029322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575801.477380:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575802.053297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575802.501386:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575803.077339:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575803.525388:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575803.589305:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575803.589315:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575803.589320:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575803.589331:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6951 00000100:00080000:2.0:1788575803.589340:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575803.589348:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575803.589383:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488942848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575803.589413:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575803.589416:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575803.589417:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575803.589420:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6951 00000100:00080000:2.0:1788575803.589422:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575803.589425:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575803.589432:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575803.589442:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488942976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575803.589449:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575803.589452:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575803.589453:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575803.589456:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6951 00000100:00080000:2.0:1788575803.589458:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788575803.589460:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488942848:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788575803.589464:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575803.589469:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575803.589476:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488943104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575803.589482:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575803.589485:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575803.589486:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575803.589488:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6951 00000100:00080000:2.0:1788575803.589491:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575803.589493:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575803.589497:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575803.589502:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60380 x1875450488943232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575803.589508:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575803.589509:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575803.589511:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575803.589516:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6951 00000800:00000100:3.0:1788575803.589516:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788575803.589520:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575803.589524:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575803.589528:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575803.589542:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60700 x1875450488943360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575803.589549:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575803/real 1788575803] req@ffff94d408c61180 x1875450488942848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575803.589583:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488942848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575803.589592:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575803.589596:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575803.589600:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488942848:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575803.589609:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488942976:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575803.589643:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575803.589653:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575803/real 1788575803] req@ffff94d408c62d80 x1875450488942976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575803.589660:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488942976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575803.589665:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575803.589667:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575803.589669:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488942976:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575803.589685:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488943104:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575803.589702:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575803.589712:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575803/real 1788575803] req@ffff94d408c60000 x1875450488943104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575803.589724:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488943104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575803.589732:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575803.589735:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575803.589738:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488943104:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575803.589743:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488943232:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575803.589761:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575803.589771:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575803/real 1788575803] req@ffff94d408c60380 x1875450488943232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575803.589780:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60380 x1875450488943232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575803.589784:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575803.589786:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575803.589788:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488943232:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575803.589792:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488943360:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575803.589803:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575803.589809:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575803/real 1788575803] req@ffff94d408c60700 x1875450488943360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575803.589815:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60700 x1875450488943360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575803.589820:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575803.589829:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575803.589832:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488943360:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575804.101323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575804.549377:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575805.125308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575805.573409:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575806.149347:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575806.597383:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575807.173351:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575807.621398:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575808.197329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575808.645362:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575808.709333:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575808.709344:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575808.709349:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575808.709361:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6957 00000100:00080000:1.0:1788575808.709375:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575808.709384:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575808.709451:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488943488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575808.709484:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575808.709489:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575808.709491:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575808.709495:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6957 00000100:00080000:1.0:1788575808.709499:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575808.709504:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575808.709515:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575808.709530:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488943616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575808.709542:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488943488:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788575808.709547:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575808.709557:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575808.709560:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575808.709564:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6957 00000100:00080000:1.0:1788575808.709568:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575808.709572:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575808.709577:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575808.709587:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488943744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575808.709594:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575808.709597:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575808.709599:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575808.709603:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6957 00000100:00080000:1.0:1788575808.709606:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575808.709608:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575808.709613:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575808.709619:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488943872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575808.709625:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575808.709630:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575808.709635:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788575808.709635:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788575808.709646:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6957 00000100:00080000:1.0:1788575808.709649:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575808.709653:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575808.709655:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575808.709665:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488944000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575808.709707:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575808/real 1788575808] req@ffff94d4049bc000 x1875450488943488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575808.709728:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488943488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575808.709739:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575808.709744:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575808.709759:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488943488:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575808.709790:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488943616:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575808.709814:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575808.709826:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575808/real 1788575808] req@ffff94d4049bfb80 x1875450488943616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575808.709839:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488943616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575808.709847:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575808.709851:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575808.709854:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488943616:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575808.709860:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488943744:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575808.709879:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575808.709890:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575808/real 1788575808] req@ffff94d4049bd180 x1875450488943744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575808.709901:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488943744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575808.709908:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575808.709912:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575808.709914:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488943744:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575808.709921:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488943872:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575808.709939:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575808.709948:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575808/real 1788575808] req@ffff94d4049bce00 x1875450488943872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575808.709958:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488943872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575808.709965:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575808.709968:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575808.709971:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488943872:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575808.709977:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488944000:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575808.709992:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575808.710000:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575808/real 1788575808] req@ffff94d4049bea00 x1875450488944000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575808.710011:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488944000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575808.710018:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575808.710021:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575808.710024:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488944000:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575809.221350:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575809.669435:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575810.245335:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575810.693385:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575811.269355:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575811.717580:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575812.293305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575812.741421:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575813.317382:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575813.765424:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575813.829403:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575813.829419:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575813.829424:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575813.829437:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6962 00000100:00080000:2.0:1788575813.829452:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575813.829462:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575813.829511:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488944128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575813.829553:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575813.829557:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575813.829559:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575813.829565:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6962 00000100:00080000:2.0:1788575813.829569:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575813.829573:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575813.829584:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575813.829597:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488944256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575813.829606:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575813.829609:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575813.829611:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575813.829614:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6962 00000100:00080000:2.0:1788575813.829618:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575813.829621:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575813.829626:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575813.829637:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62300 x1875450488944384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575813.829646:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575813.829648:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575813.829650:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575813.829653:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6962 00000100:00080000:2.0:1788575813.829656:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575813.829659:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575813.829663:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575813.829672:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60700 x1875450488944512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575813.829679:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575813.829683:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575813.829684:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575813.829687:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6962 00000100:00080000:2.0:1788575813.829689:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575813.829693:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575813.829696:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575813.829704:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60380 x1875450488944640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575813.829806:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488944128:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575813.829911:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488944256:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575813.829936:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488944384:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575813.829956:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488944512:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575813.829975:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488944640:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575814.341324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575814.789441:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575815.365378:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575815.813347:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575816.389316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575816.837375:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788575816.901524:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:1.0:1788575816.901573:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788575816.901604:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575816.901639:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575816.901648:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575816.901659:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575816.901669:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:3.0:1788575816.901675:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575813/real 1788575816] req@ffff94d408c63800 x1875450488944128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575816.901704:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488944128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575816.901715:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575816.901720:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575816.901725:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488944128:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575816.901733:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575813/real 1788575816] req@ffff94d408c62680 x1875450488944256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575816.901745:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488944256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575816.901753:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575816.901756:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575816.901759:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488944256:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575816.901766:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575813/real 1788575816] req@ffff94d408c62300 x1875450488944384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575816.901777:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62300 x1875450488944384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575816.901784:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575816.901787:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575816.901790:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488944384:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575816.901795:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575813/real 1788575816] req@ffff94d408c60700 x1875450488944512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575816.901806:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60700 x1875450488944512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575816.901813:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575816.901816:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575816.901819:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488944512:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575816.901826:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575813/real 1788575816] req@ffff94d408c60380 x1875450488944640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575816.901836:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60380 x1875450488944640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575816.901843:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575816.901847:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575816.901849:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488944640:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575817.413306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575817.861347:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575818.437311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575818.885337:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575818.949361:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575818.949372:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575818.949376:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575818.949387:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6967 00000100:00080000:2.0:1788575818.949398:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575818.949406:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575818.949445:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488944768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575818.949471:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575818.949475:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575818.949476:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575818.949480:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6967 00000100:00080000:2.0:1788575818.949484:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575818.949488:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575818.949496:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575818.949506:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488944896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575818.949512:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575818.949515:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575818.949516:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575818.949518:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6967 00000100:00080000:2.0:1788575818.949520:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575818.949522:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575818.949525:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575818.949530:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488945024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575818.949536:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575818.949538:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575818.949539:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575818.949541:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6967 00000100:00080000:2.0:1788575818.949542:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575818.949545:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575818.949548:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575818.949556:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488945152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575818.949564:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575818.949572:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575818.949574:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575818.949577:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6967 00000100:00080000:2.0:1788575818.949581:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575818.949584:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575818.949588:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575818.949598:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488945280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575818.949779:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488944768:192.168.202.125@tcp:250:kworker.0 00000800:00000100:3.0:1788575818.949850:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575818.949885:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575818/real 1788575818] req@ffff94d408c60000 x1875450488944768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575818.949901:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488944768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575818.949916:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575818.949920:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575818.949923:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488944768:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575818.949930:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488944896:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575818.949952:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575818.949964:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575818/real 1788575818] req@ffff94d408c62d80 x1875450488944896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575818.949975:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488944896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575818.949982:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575818.949986:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575818.949989:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488944896:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575818.949995:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488945024:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575818.950013:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575818.950023:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575818/real 1788575818] req@ffff94d408c61180 x1875450488945024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575818.950040:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488945024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575818.950047:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575818.950149:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575818.950153:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488945024:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575818.950160:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488945152:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575818.950183:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575818.950194:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575818/real 1788575818] req@ffff94d408c63480 x1875450488945152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575818.950234:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488945152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575818.950241:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575818.950244:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575818.950246:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488945152:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575818.950253:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488945280:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575818.950271:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575818.950281:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575818/real 1788575818] req@ffff94d408c61f80 x1875450488945280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575818.950292:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488945280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575818.951638:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575818.951644:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575818.951647:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488945280:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575819.461269:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575819.909323:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575820.485301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575820.933387:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575821.509317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575821.957372:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575822.533480:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575822.981345:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575823.557573:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575824.005342:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575824.069336:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575824.069347:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575824.069350:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575824.069359:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6972 00000100:00080000:1.0:1788575824.069367:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575824.069374:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575824.069402:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488945408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575824.069426:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575824.069429:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575824.069431:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575824.069434:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6972 00000100:00080000:1.0:1788575824.069436:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575824.069439:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575824.069446:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575824.069455:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488945536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575824.069463:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575824.069465:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575824.069466:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575824.069471:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6972 00000100:00100000:2.0:1788575824.069472:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488945408:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788575824.069474:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575824.069482:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575824.069488:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575824.069502:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488945664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575824.069512:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575824.069524:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575824.069526:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575824.069530:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6972 00000100:00080000:1.0:1788575824.069533:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788575824.069536:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788575824.069540:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575824.069545:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575824.069555:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488945792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575824.069562:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788575824.069570:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575824/real 1788575824] req@ffff94d4049bc380 x1875450488945408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575824.069573:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575824.069577:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575824.069587:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6972 00000100:00080000:1.0:1788575824.069592:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575824.069597:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575824.069600:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc380 x1875450488945408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788575824.069604:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788575824.069616:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1788575824.069621:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488945920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575824.069630:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575824.069639:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488945408:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575824.069677:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488945536:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575824.069698:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575824.069711:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575824/real 1788575824] req@ffff94d4049bf480 x1875450488945536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575824.069723:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488945536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575824.069732:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575824.069743:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575824.069747:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488945536:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575824.069754:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488945664:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575824.069776:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575824.069790:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575824/real 1788575824] req@ffff94d4049bea00 x1875450488945664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575824.069802:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488945664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575824.069810:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575824.069814:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575824.069818:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488945664:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575824.069825:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488945792:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575824.069846:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575824.069859:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575824/real 1788575824] req@ffff94d4049bce00 x1875450488945792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575824.069874:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488945792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575824.069882:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575824.069886:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575824.069889:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488945792:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575824.069896:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488945920:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575824.069917:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575824.069930:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575824/real 1788575824] req@ffff94d4049bd180 x1875450488945920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575824.069948:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488945920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575824.069955:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575824.069958:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575824.069961:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488945920:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575824.581784:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575825.029454:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575825.607181:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575826.053464:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575826.629341:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575827.077411:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575827.654497:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575828.101537:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575828.677298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575829.125385:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575829.189322:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575829.189329:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575829.189333:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575829.189340:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6977 00000100:00080000:3.0:1788575829.189349:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575829.189354:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575829.189383:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488946048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575829.189401:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575829.189403:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575829.189404:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575829.189406:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6977 00000100:00080000:3.0:1788575829.189408:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575829.189410:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575829.189417:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575829.189424:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488946176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575829.189431:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575829.189435:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788575829.189436:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488946048:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575829.189437:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575829.189444:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6977 00000100:00080000:3.0:1788575829.189447:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575829.189449:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575829.189451:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575829.189458:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488946304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575829.189463:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575829.189466:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575829.189467:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575829.189469:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6977 00000100:00080000:3.0:1788575829.189484:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575829.189487:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575829.189490:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788575829.189494:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575829.189498:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488946432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575829.189505:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575829.189507:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575829.189509:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575829.189511:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6977 00000100:00080000:3.0:1788575829.189514:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575829.189516:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575829.189519:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575829.189528:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575829/real 1788575829] req@ffff94d408c55f80 x1875450488946048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575829.189528:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488946560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575829.189562:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488946048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575829.189573:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575829.189577:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575829.189581:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488946048:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575829.189602:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488946176:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575829.189625:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575829.189637:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575829/real 1788575829] req@ffff94d408c54700 x1875450488946176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575829.189650:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488946176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575829.189658:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575829.189662:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575829.189665:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488946176:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575829.189670:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488946304:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575829.189689:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575829.189701:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575829/real 1788575829] req@ffff94d408c56300 x1875450488946304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575829.189711:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488946304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575829.189717:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575829.189719:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575829.189721:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488946304:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575829.189727:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488946432:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575829.189741:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575829.189747:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575829/real 1788575829] req@ffff94d408c55880 x1875450488946432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575829.189757:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488946432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575829.189766:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575829.189768:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575829.189769:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488946432:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575829.189773:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488946560:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575829.189791:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575829.189797:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575829/real 1788575829] req@ffff94d408c55180 x1875450488946560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575829.189807:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488946560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575829.189812:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575829.189813:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575829.189815:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488946560:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575829.701324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575830.149328:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575830.725305:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575831.173380:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575831.749260:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575832.197360:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575832.773312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575833.221396:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575833.797263:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575834.245370:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575834.309327:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575834.309334:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575834.309338:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575834.309346:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6982 00000100:00080000:3.0:1788575834.309354:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575834.309359:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575834.309390:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488946688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575834.309414:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575834.309418:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575834.309419:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575834.309423:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6982 00000100:00080000:3.0:1788575834.309426:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575834.309429:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575834.309438:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575834.309459:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488946816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575834.309467:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488946688:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575834.309473:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575834.309477:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575834.309478:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575834.309481:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6982 00000100:00080000:3.0:1788575834.309484:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575834.309487:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575834.309491:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575834.309501:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488946944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575834.309507:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575834.309509:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575834.309510:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575834.309512:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6982 00000100:00080000:3.0:1788575834.309517:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575834.309519:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575834.309521:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788575834.309527:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575834.309529:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488947072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575834.309548:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575834.309552:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575834.309554:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575834.309558:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6982 00000100:00100000:2.0:1788575834.309563:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575834/real 1788575834] req@ffff94d408c57480 x1875450488946688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575834.309563:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575834.309571:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575834.309581:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575834.309587:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488946688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575834.309594:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488947200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575834.309601:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575834.309609:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575834.309613:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488946688:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575834.309637:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488946816:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575834.309655:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575834.309666:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575834/real 1788575834] req@ffff94d408c54000 x1875450488946816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575834.309676:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488946816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575834.309683:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575834.309686:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575834.309689:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488946816:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575834.309693:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488946944:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575834.309721:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575834.309730:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575834/real 1788575834] req@ffff94d408c55180 x1875450488946944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575834.309739:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488946944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575834.309747:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575834.309750:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575834.309752:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488946944:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575834.309758:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488947072:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575834.309774:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575834.309782:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575834/real 1788575834] req@ffff94d408c55880 x1875450488947072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575834.309792:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488947072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575834.309799:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575834.309802:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575834.309805:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488947072:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575834.309809:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488947200:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575834.309824:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575834.309833:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575834/real 1788575834] req@ffff94d408c56300 x1875450488947200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575834.309842:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488947200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575834.309849:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575834.309852:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575834.309855:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488947200:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575834.821333:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575835.269334:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575835.845258:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575836.294046:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575836.869317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575837.317377:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575837.893307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575838.341393:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575838.917287:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575839.365380:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575839.429337:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575839.429346:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575839.429349:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575839.429357:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6987 00000100:00080000:2.0:1788575839.429367:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575839.429373:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575839.429402:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488947328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575839.429424:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575839.429426:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575839.429427:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575839.429430:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6987 00000100:00080000:2.0:1788575839.429432:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575839.429434:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575839.429440:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575839.429447:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488947456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575839.429453:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575839.429455:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575839.429456:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575839.429458:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6987 00000100:00080000:2.0:1788575839.429459:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575839.429461:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575839.429464:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575839.429469:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488947584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575839.429475:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575839.429476:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575839.429477:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575839.429479:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6987 00000100:00080000:2.0:1788575839.429481:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575839.429482:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575839.429484:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575839.429489:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488947712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575839.429494:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575839.429496:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575839.429497:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575839.429500:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6987 00000100:00080000:2.0:1788575839.429502:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575839.429505:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575839.429508:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575839.429515:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488947840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575839.429691:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488947328:192.168.202.125@tcp:250:kworker.0 00000800:00000100:3.0:1788575839.429749:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575839.429783:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575839/real 1788575839] req@ffff94d408c62a00 x1875450488947328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575839.429801:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488947328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575839.429812:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575839.429815:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575839.429819:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488947328:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575839.429826:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488947456:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575839.429856:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575839.429867:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575839/real 1788575839] req@ffff94d408c63b80 x1875450488947456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575839.429879:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488947456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575839.429887:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575839.429890:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575839.429894:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488947456:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575839.429899:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488947584:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575839.429919:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575839.429930:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575839/real 1788575839] req@ffff94d408c60a80 x1875450488947584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575839.429940:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488947584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575839.429948:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575839.429951:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575839.429955:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488947584:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575839.429960:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488947712:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575839.429977:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575839.429986:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575839/real 1788575839] req@ffff94d408c61500 x1875450488947712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575839.429998:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488947712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575839.430006:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575839.430009:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575839.430012:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488947712:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575839.430028:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488947840:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575839.430048:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575839.430060:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575839/real 1788575839] req@ffff94d408c61880 x1875450488947840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575839.430072:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488947840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575839.430079:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575839.430082:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575839.430085:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488947840:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575839.941316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575840.389479:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575840.965275:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575841.413422:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575841.989330:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575842.437400:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575843.013308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575843.461384:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575844.038570:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575844.485385:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575844.549303:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575844.549310:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575844.549314:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575844.549321:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6992 00000100:00080000:2.0:1788575844.549331:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575844.549337:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575844.549366:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63100 x1875450488947968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575844.549389:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575844.549392:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575844.549393:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575844.549395:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6992 00000100:00080000:2.0:1788575844.549397:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575844.549399:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575844.549406:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575844.549429:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60e00 x1875450488948096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575844.549437:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575844.549440:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575844.549441:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575844.549444:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6992 00000100:00080000:2.0:1788575844.549447:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788575844.549451:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488947968:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788575844.549453:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575844.549461:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575844.549469:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488948224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575844.549475:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575844.549478:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575844.549479:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575844.549481:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6992 00000100:00080000:2.0:1788575844.549483:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575844.549486:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575844.549489:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575844.549500:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488948352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575844.549508:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575844.549511:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575844.549514:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788575844.549516:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788575844.549520:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6992 00000100:00080000:2.0:1788575844.549524:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575844.549529:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575844.549534:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575844.549545:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488948480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575844.549555:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575844/real 1788575844] req@ffff94d408c63100 x1875450488947968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575844.549572:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63100 x1875450488947968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575844.549581:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575844.549585:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575844.549589:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488947968:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575844.549596:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488948096:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575844.549617:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575844.549627:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575844/real 1788575844] req@ffff94d408c60e00 x1875450488948096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575844.549639:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60e00 x1875450488948096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575844.549644:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575844.549645:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575844.549647:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488948096:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575844.549664:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488948224:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575844.549676:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575844.549681:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575844/real 1788575844] req@ffff94d408c61c00 x1875450488948224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575844.549688:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488948224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575844.549692:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575844.549694:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575844.549695:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488948224:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575844.549698:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488948352:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575844.549707:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575844.549712:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575844/real 1788575844] req@ffff94d408c61880 x1875450488948352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575844.549718:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488948352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575844.549729:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575844.549732:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575844.549735:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488948352:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575844.549739:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488948480:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575844.549755:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575844.549766:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575844/real 1788575844] req@ffff94d408c61500 x1875450488948480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575844.549777:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488948480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575844.549783:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575844.549785:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575844.549787:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488948480:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575845.061334:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575845.509444:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575846.085336:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575846.533397:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575847.109326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575847.557473:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575848.133318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575848.581425:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575849.157337:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575849.605433:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575849.670402:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575849.670413:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575849.670418:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575849.670428:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 6997 00000100:00080000:3.0:1788575849.670438:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575849.670506:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575849.670542:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488948608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575849.670610:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488948608:192.168.202.125@tcp:250:kworker.0 00000800:00000100:2.0:1788575849.670676:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575849.670704:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575849.670708:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575849.670710:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575849.670715:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6997 00000100:00080000:3.0:1788575849.670718:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575849.670722:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575849.670732:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575849.670744:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488948736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575849.670849:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575849/real 1788575849] req@ffff94d408c54700 x1875450488948608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575849.670866:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488948608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575849.670875:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575849.670887:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575849.670891:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488948608:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575849.670931:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488948736:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575849.670953:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575849.670966:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575849/real 1788575849] req@ffff94d408c55f80 x1875450488948736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575849.670977:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488948736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575849.670984:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575849.670987:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575849.670990:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488948736:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788575849.671317:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575849.671322:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575849.671323:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575849.671328:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6997 00000100:00080000:3.0:1788575849.671333:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575849.671337:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575849.671342:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575849.671354:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488948864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575849.671397:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488948864:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575849.671419:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575849.671440:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575849.671444:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575849.671445:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575849.671457:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6997 00000100:00080000:3.0:1788575849.671461:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575849.671466:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575849.671471:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575849.671483:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488948992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575849.671564:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575849/real 1788575849] req@ffff94d408c55c00 x1875450488948864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575849.671577:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488948864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575849.671586:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575849.671589:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575849.671592:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488948864:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575849.671604:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488948992:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575849.671626:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575849.671637:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575849/real 1788575849] req@ffff94d408c54e00 x1875450488948992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575849.671647:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488948992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575849.671712:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575849.671716:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575849.671720:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488948992:192.168.202.125@tcp:8:kworker.0 00000100:00080000:3.0:1788575849.671803:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575849.671807:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575849.671809:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575849.671813:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 6997 00000100:00080000:3.0:1788575849.671817:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575849.671822:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575849.671826:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575849.671940:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488949120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575849.671985:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488949120:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575849.672008:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575849.672288:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575849/real 1788575849] req@ffff94d408c56a00 x1875450488949120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575849.672320:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488949120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575849.672328:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575849.672331:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575849.672335:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488949120:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575850.181392:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575850.629430:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575851.205335:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575851.653389:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575852.229330:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575852.677412:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575853.253356:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575853.701386:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575854.277311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575854.725320:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575854.789302:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575854.789312:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575854.789316:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575854.789326:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7003 00000100:00080000:1.0:1788575854.789336:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575854.789344:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575854.789377:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488949248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575854.789401:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575854.789403:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575854.789404:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575854.789407:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7003 00000100:00080000:1.0:1788575854.789409:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575854.789412:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575854.789419:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575854.789429:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488949376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575854.789431:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488949248:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788575854.789443:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575854.789445:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575854.789446:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575854.789448:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7003 00000100:00080000:1.0:1788575854.789449:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575854.789451:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575854.789454:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575854.789460:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488949504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575854.789466:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575854.789467:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575854.789469:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575854.789471:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7003 00000100:00080000:1.0:1788575854.789473:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575854.789475:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575854.789479:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575854.789485:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488949632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788575854.789489:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788575854.789494:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575854.789497:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575854.789498:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575854.789501:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7003 00000100:00080000:1.0:1788575854.789503:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575854.789506:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575854.789508:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575854.789513:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488949760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575854.789532:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575854/real 1788575854] req@ffff94d4049bfb80 x1875450488949248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575854.789549:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488949248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575854.789556:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575854.789559:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575854.789562:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488949248:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575854.789578:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488949376:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575854.789594:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575854.789601:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575854/real 1788575854] req@ffff94d4049bc000 x1875450488949376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575854.789609:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488949376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575854.789614:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575854.789616:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575854.789633:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488949376:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575854.789637:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488949504:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575854.789652:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575854.789663:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575854/real 1788575854] req@ffff94d4049bc700 x1875450488949504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575854.789676:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488949504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575854.789684:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575854.789688:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575854.789691:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488949504:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575854.789697:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488949632:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575854.789714:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575854.789721:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575854/real 1788575854] req@ffff94d4049bed80 x1875450488949632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575854.789727:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488949632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575854.789739:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575854.789742:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575854.789745:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488949632:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575854.789750:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488949760:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575854.789766:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575854.789776:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575854/real 1788575854] req@ffff94d4049bf800 x1875450488949760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575854.789787:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488949760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575854.789795:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575854.789798:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575854.789801:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488949760:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575855.301311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575855.749423:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575856.325275:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575856.773395:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575857.350075:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575857.797325:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575858.373324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575858.823284:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575859.397312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575859.845355:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575859.909386:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575859.909394:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575859.909398:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575859.909413:0:66171:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7008 00000100:00080000:1.0:1788575859.909424:0:66171:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575859.909431:0:66171:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575859.909464:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488949888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575859.909490:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575859.909496:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575859.909498:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575859.909502:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7008 00000100:00080000:1.0:1788575859.909506:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575859.909511:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575859.909520:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575859.909532:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488949888:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575859.909535:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488950016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575859.909547:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575859.909550:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575859.909552:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575859.909555:0:66171:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7008 00000100:00080000:1.0:1788575859.909559:0:66171:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575859.909563:0:66171:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575859.909568:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575859.909580:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488950144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575859.909584:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788575859.909594:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575859.909598:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575859.909600:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575859.909605:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7008 00000100:00080000:1.0:1788575859.909610:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575859.909615:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575859.909620:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575859.909623:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575859/real 1788575859] req@ffff94d4049be300 x1875450488949888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575859.909634:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488950272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575859.909644:0:66171:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575859.909645:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488949888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575859.909651:0:66171:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575859.909655:0:66171:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575859.909662:0:66171:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7008 00000100:00080000:1.0:1788575859.909667:0:66171:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575859.909674:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575859.909680:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:1.0:1788575859.909682:0:66171:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575859.909685:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488949888:192.168.202.125@tcp:250:kworker.0 00000008:00100000:1.0:1788575859.909689:0:66171:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575859.909710:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488950016:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575859.909717:0:66171:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488950400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575859.909737:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575859.909751:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575859/real 1788575859] req@ffff94d4049bf100 x1875450488950016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575859.909759:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488950016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575859.909765:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575859.909767:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575859.909770:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488950016:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575859.909776:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488950144:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575859.909796:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575859.909807:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575859/real 1788575859] req@ffff94d4049be680 x1875450488950144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575859.909820:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488950144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575859.909827:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575859.909830:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575859.909832:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488950144:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575859.909836:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488950272:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575859.909852:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575859.909860:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575859/real 1788575859] req@ffff94d4049bdc00 x1875450488950272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575859.909867:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488950272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575859.909876:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575859.909880:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575859.909885:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488950272:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575859.909910:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488950400:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575859.909930:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575859.909941:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575859/real 1788575859] req@ffff94d4049bd500 x1875450488950400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575859.909950:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488950400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575859.909956:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575859.909958:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575859.909960:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488950400:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575860.421327:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575860.869426:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575861.445377:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575861.893379:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575862.469359:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575862.917413:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575863.493316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575863.941336:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575864.517304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575864.966432:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575865.029305:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575865.029315:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575865.029319:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575865.029328:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7013 00000100:00080000:1.0:1788575865.029338:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575865.029344:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575865.029380:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488950528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575865.029407:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575865.029411:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575865.029412:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575865.029416:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7013 00000100:00080000:1.0:1788575865.029420:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575865.029423:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575865.029432:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575865.029445:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd880 x1875450488950656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575865.029456:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575865.029459:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575865.029463:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575865.029464:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488950528:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788575865.029469:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7013 00000100:00080000:1.0:1788575865.029486:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575865.029490:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575865.029495:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575865.029508:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488950784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575865.029517:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575865.029520:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575865.029522:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575865.029527:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7013 00000800:00000100:0.0:1788575865.029528:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788575865.029532:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575865.029536:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575865.029541:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575865.029550:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488950912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575865.029564:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575865/real 1788575865] req@ffff94d4049bca80 x1875450488950528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575865.029568:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575865.029576:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575865.029579:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575865.029585:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7013 00000100:00080000:1.0:1788575865.029588:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788575865.029593:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488950528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575865.029595:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575865.029603:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788575865.029609:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575865.029616:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575865.029616:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488951040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575865.029624:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488950528:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575865.029653:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488950656:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575865.029675:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575865.029688:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575865/real 1788575865] req@ffff94d4049bd880 x1875450488950656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575865.029700:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd880 x1875450488950656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575865.029708:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575865.029711:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575865.029714:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488950656:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575865.029720:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488950784:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575865.029737:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575865.029747:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575865/real 1788575865] req@ffff94d4049bd500 x1875450488950784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575865.029764:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488950784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575865.029772:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575865.029775:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575865.029778:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488950784:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575865.029783:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488950912:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575865.029805:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575865.029820:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575865/real 1788575865] req@ffff94d4049bdc00 x1875450488950912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575865.029830:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488950912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575865.029837:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575865.029840:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575865.029843:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488950912:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575865.029849:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488951040:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575865.029864:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575865.029874:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575865/real 1788575865] req@ffff94d4049be680 x1875450488951040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575865.029885:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488951040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575865.029899:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575865.029902:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575865.029904:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488951040:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575865.541377:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575865.989405:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575866.565303:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575867.013537:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575867.589328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575868.037375:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575868.613329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575869.061373:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575869.637322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575870.085375:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575870.149324:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575870.149332:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575870.149335:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575870.149343:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7018 00000100:00080000:3.0:1788575870.149351:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575870.149357:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575870.149398:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488951168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575870.149425:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575870.149428:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575870.149429:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575870.149432:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7018 00000100:00080000:3.0:1788575870.149434:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575870.149437:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575870.149444:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575870.149454:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488951296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575870.149461:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575870.149464:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575870.149465:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575870.149467:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7018 00000100:00080000:3.0:1788575870.149469:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575870.149474:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575870.149476:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488951168:192.168.202.125@tcp:250:kworker.0 00000008:00100000:3.0:1788575870.149478:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575870.149486:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488951424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575870.149532:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575870.149544:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575870.149547:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575870.149548:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575870.149552:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7018 00000100:00080000:3.0:1788575870.149556:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575870.149559:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575870.149563:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575870.149573:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488951552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575870.149578:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575870.149581:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575870/real 1788575870] req@ffff94d408c54380 x1875450488951168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575870.149581:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575870.149583:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575870.149589:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7018 00000100:00080000:3.0:1788575870.149594:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575870.149600:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575870.149605:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575870.149610:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488951168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575870.149617:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488951680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575870.149624:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575870.149631:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575870.149636:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488951168:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575870.149644:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488951296:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575870.149663:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575870.149674:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575870/real 1788575870] req@ffff94d408c57800 x1875450488951296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575870.149684:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488951296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575870.149698:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575870.149701:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575870.149703:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488951296:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575870.149723:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488951424:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575870.149738:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575870.149747:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575870/real 1788575870] req@ffff94d408c57b80 x1875450488951424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575870.149757:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488951424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575870.149765:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575870.149768:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575870.149771:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488951424:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575870.149776:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488951552:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575870.149791:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575870.149800:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575870/real 1788575870] req@ffff94d408c56d80 x1875450488951552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575870.149810:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488951552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575870.149817:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575870.149820:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575870.149823:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488951552:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575870.149828:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488951680:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575870.149843:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575870.149852:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575870/real 1788575870] req@ffff94d408c56680 x1875450488951680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575870.149861:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488951680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575870.149868:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575870.149871:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575870.149886:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488951680:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575870.661289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575871.109397:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575871.685296:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575872.133404:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575872.709284:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575873.157387:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575873.733266:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575874.181386:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575874.757315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575875.205370:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575875.269658:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575875.269668:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575875.269673:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575875.269682:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7023 00000100:00080000:3.0:1788575875.269692:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575875.269699:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575875.269735:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488951808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575875.269766:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575875.269781:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575875.269783:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575875.269787:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7023 00000100:00080000:3.0:1788575875.269790:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575875.269795:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575875.269803:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575875.269817:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488951936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575875.269825:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488951808:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575875.269832:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575875.269837:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575875.269838:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575875.269843:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7023 00000100:00080000:3.0:1788575875.270086:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575875.270092:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575875.270098:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788575875.270106:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575875.270114:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488952064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575875.270125:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575875.270128:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575875.270130:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575875.270135:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7023 00000100:00080000:3.0:1788575875.270139:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575875.270144:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575875.270149:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575875.270154:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575875/real 1788575875] req@ffff94d408c54a80 x1875450488951808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575875.270163:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488952192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575875.270175:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575875.270181:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488951808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575875.270181:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575875.270185:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575875.270194:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7023 00000100:00080000:0.0:1788575875.270199:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575875.270200:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575875.270224:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575875.270229:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488951808:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575875.270239:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575875.270245:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575875.270258:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488952320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575875.270263:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488951936:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575875.270298:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575875.270475:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575875/real 1788575875] req@ffff94d408c57100 x1875450488951936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575875.270489:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488951936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575875.270498:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575875.270501:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575875.270505:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488951936:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575875.270510:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488952064:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575875.270537:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575875.270548:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575875/real 1788575875] req@ffff94d408c56680 x1875450488952064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575875.270560:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488952064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575875.270568:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575875.270571:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575875.270574:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488952064:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575875.270580:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488952192:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575875.270600:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575875.270619:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575875/real 1788575875] req@ffff94d408c56d80 x1875450488952192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575875.270627:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488952192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575875.270631:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575875.270633:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575875.270635:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488952192:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575875.270646:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488952320:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575875.270670:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575875.270681:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575875/real 1788575875] req@ffff94d408c57b80 x1875450488952320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575875.270693:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488952320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575875.270700:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575875.270703:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575875.270716:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488952320:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575875.781302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575876.229381:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575876.805309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575877.253420:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575877.829416:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575878.277382:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575878.853307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575879.301424:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575879.877367:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575880.325429:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575880.389319:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575880.389327:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575880.389330:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575880.389339:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7028 00000100:00080000:3.0:1788575880.389348:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575880.389357:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575880.389394:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488952448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575880.389424:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575880.389429:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575880.389431:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575880.389435:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7028 00000100:00080000:3.0:1788575880.389438:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575880.389442:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575880.389451:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575880.389466:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488952576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575880.389477:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488952448:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575880.389479:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575880.389486:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575880.389488:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575880.389493:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7028 00000100:00080000:3.0:1788575880.389497:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575880.389501:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575880.389506:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575880.389519:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488952704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575880.389528:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575880.389531:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575880.389532:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575880.389536:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7028 00000100:00080000:3.0:1788575880.389539:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575880.389542:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575880.389547:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575880.389558:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488952832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575880.389568:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575880.389569:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488952576:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788575880.389575:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575880.389579:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575880.389585:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7028 00000100:00080000:3.0:1788575880.389589:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575880.389594:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575880.389600:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575880.389603:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488952704:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575880.389616:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488952960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575880.389629:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488952832:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575880.389652:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488952960:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575880.901357:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575881.349468:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575881.925371:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575882.373424:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575882.949383:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575883.397427:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788575883.462165:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:1.0:1788575883.462256:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788575883.462272:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575883.462299:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575883.462306:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575883.462312:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788575883.462319:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:0.0:1788575883.462406:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575880/real 1788575883] req@ffff94d408c54e00 x1875450488952832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575883.462431:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488952832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575883.462442:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575883.462446:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575883.462450:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488952832:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575883.462458:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575880/real 1788575883] req@ffff94d408c55c00 x1875450488952960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575883.462468:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488952960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575883.462476:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575883.462479:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575883.462482:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488952960:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575883.462488:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575880/real 1788575883] req@ffff94d408c54380 x1875450488952576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575883.462499:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488952576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575883.462506:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575883.462509:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575883.462512:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488952576:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575883.462518:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575880/real 1788575883] req@ffff94d408c56a00 x1875450488952704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575883.462527:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488952704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575883.462535:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575883.462538:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575883.462540:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488952704:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575883.462546:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575880/real 1788575883] req@ffff94d408c57800 x1875450488952448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575883.462556:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488952448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575883.462562:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575883.462565:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575883.462568:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488952448:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575883.973388:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575884.421351:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575884.997306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575885.445382:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575885.509371:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575885.509381:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575885.509384:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575885.509393:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7033 00000100:00080000:1.0:1788575885.509403:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575885.509410:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575885.509441:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488953088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575885.509466:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575885.509470:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575885.509472:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575885.509476:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7033 00000100:00080000:1.0:1788575885.509479:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575885.509483:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575885.509492:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575885.509507:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488953088:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575885.509508:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488953216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575885.509522:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575885.509525:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575885.509526:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575885.509529:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7033 00000100:00080000:1.0:1788575885.509531:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575885.509534:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575885.509538:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575885.509548:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488953344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575885.509553:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575885.509556:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575885.509557:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575885.509571:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7033 00000100:00080000:1.0:1788575885.509575:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575885.509579:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575885.509584:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788575885.509598:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575885.509598:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488953472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575885.509609:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575885.509612:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575885.509614:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575885.509618:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7033 00000100:00080000:1.0:1788575885.509622:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575885.509628:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575885.509633:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575885.509642:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488953600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575885.509651:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575885/real 1788575885] req@ffff94d4049bf100 x1875450488953088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575885.509668:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488953088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575885.509678:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575885.509683:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575885.509687:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488953088:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575885.509708:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488953216:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575885.509731:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575885.509743:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575885/real 1788575885] req@ffff94d4049be300 x1875450488953216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575885.509753:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488953216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575885.509761:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575885.509764:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575885.509767:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488953216:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575885.509772:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488953344:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575885.509791:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575885.509802:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575885/real 1788575885] req@ffff94d4049bf800 x1875450488953344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575885.509812:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488953344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575885.509824:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575885.509828:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575885.509831:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488953344:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575885.509836:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488953472:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575885.509855:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575885.509865:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575885/real 1788575885] req@ffff94d4049bed80 x1875450488953472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575885.509876:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488953472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575885.509884:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575885.509887:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575885.509890:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488953472:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575885.509896:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488953600:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575885.509914:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575885.509925:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575885/real 1788575885] req@ffff94d4049bc700 x1875450488953600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575885.509935:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488953600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575885.509942:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575885.509944:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575885.509947:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488953600:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575886.021323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575886.469382:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575887.045328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575887.493384:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575888.069311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575888.517338:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575889.093589:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575889.541344:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575890.117359:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575890.565361:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575890.629324:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575890.629332:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575890.629335:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575890.629344:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7038 00000100:00080000:1.0:1788575890.629353:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575890.629359:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575890.629396:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488953728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575890.629417:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575890.629423:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575890.629424:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575890.629428:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7038 00000100:00080000:1.0:1788575890.629431:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575890.629436:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575890.629445:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575890.629470:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488953856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575890.629476:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488953728:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788575890.629481:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575890.629485:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575890.629486:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575890.629489:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7038 00000100:00080000:1.0:1788575890.629492:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575890.629495:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575890.629504:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575890.629515:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488953984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575890.629521:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575890.629523:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575890.629524:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575890.629526:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7038 00000100:00080000:1.0:1788575890.629529:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575890.629531:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575890.629535:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788575890.629541:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575890.629542:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488954112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575890.629552:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575890.629555:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575890.629557:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575890.629562:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7038 00000100:00080000:1.0:1788575890.629565:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575890.629570:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575890.629575:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575890.629580:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575890/real 1788575890] req@ffff94d4049bc000 x1875450488953728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575890.629589:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488954240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575890.629602:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488953728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575890.629613:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575890.629617:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575890.629621:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488953728:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575890.629642:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488953856:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575890.629664:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575890.629676:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575890/real 1788575890] req@ffff94d4049bfb80 x1875450488953856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575890.629684:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488953856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575890.629690:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575890.629692:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575890.629694:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488953856:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575890.629699:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488953984:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575890.629712:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575890.629718:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575890/real 1788575890] req@ffff94d4049bd180 x1875450488953984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575890.629725:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488953984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575890.629730:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575890.629732:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575890.629734:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488953984:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575890.629737:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488954112:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575890.629746:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575890.629751:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575890/real 1788575890] req@ffff94d4049bce00 x1875450488954112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575890.629757:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488954112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575890.629762:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575890.629763:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575890.629765:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488954112:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575890.629768:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488954240:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575890.629775:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575890.629780:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575890/real 1788575890] req@ffff94d4049bea00 x1875450488954240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575890.629787:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488954240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575890.629791:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575890.629793:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575890.629794:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488954240:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575891.142178:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575891.589375:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575892.165366:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575892.613547:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575893.190011:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575893.637536:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575894.213363:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575894.661901:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575895.237379:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575895.685398:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575895.749450:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575895.749459:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575895.749465:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575895.749477:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7044 00000100:00080000:1.0:1788575895.749491:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575895.749500:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575895.749542:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488954368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575895.749577:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575895.749581:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575895.749584:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575895.749588:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7044 00000100:00080000:1.0:1788575895.749593:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575895.749597:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575895.749608:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575895.749623:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488954496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575895.749634:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575895.749637:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575895.749640:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575895.749644:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7044 00000100:00080000:1.0:1788575895.749648:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575895.749653:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575895.749658:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575895.749671:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488954624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575895.749681:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575895.749686:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1788575895.749688:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488954368:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788575895.749689:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575895.749697:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7044 00000100:00080000:1.0:1788575895.749701:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575895.749705:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575895.749710:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575895.749722:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488954752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575895.749737:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575895.749740:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575895.749742:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575895.749745:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7044 00000100:00080000:1.0:1788575895.749751:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575895.749754:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575895.749759:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575895.749769:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488954880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788575895.749783:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575895.749844:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575895/real 1788575895] req@ffff94d4049bf480 x1875450488954368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575895.749868:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488954368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575895.749882:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575895.749887:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575895.749893:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488954368:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575895.749902:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488954496:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575895.749929:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575895.749948:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575895/real 1788575895] req@ffff94d4049bc380 x1875450488954496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575895.749963:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc380 x1875450488954496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575895.749973:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575895.749976:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575895.749980:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488954496:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575895.750011:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488954624:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575895.750033:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575895.750049:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575895/real 1788575895] req@ffff94d4049bea00 x1875450488954624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575895.750061:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488954624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575895.750070:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575895.750073:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575895.750076:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488954624:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575895.750082:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488954752:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575895.750102:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575895.750113:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575895/real 1788575895] req@ffff94d4049bce00 x1875450488954752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575895.750124:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488954752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575895.750132:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575895.750135:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575895.750138:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488954752:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575895.750144:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488954880:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575895.750161:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575895.750171:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575895/real 1788575895] req@ffff94d4049bd180 x1875450488954880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575895.750182:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488954880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575895.750189:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575895.750192:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575895.750195:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488954880:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575896.261398:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575896.709496:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575897.285953:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575897.733451:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575898.309470:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575898.757375:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575899.333432:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575899.781364:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575900.357344:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575900.805991:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575900.869429:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575900.869440:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575900.869444:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575900.869454:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7049 00000100:00080000:3.0:1788575900.869466:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575900.869473:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575900.869511:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488955008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575900.869534:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575900.869538:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575900.869540:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575900.869543:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7049 00000100:00080000:3.0:1788575900.869547:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575900.869550:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575900.869560:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575900.869574:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488955136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575900.869584:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575900.869587:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575900.869589:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575900.869592:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7049 00000100:00080000:3.0:1788575900.869595:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575900.869599:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575900.869603:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575900.869613:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488955264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575900.869623:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575900.869626:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575900.869627:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575900.869631:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7049 00000100:00080000:3.0:1788575900.869634:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575900.869638:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575900.869643:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575900.869653:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488955392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575900.869663:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575900.869666:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575900.869667:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575900.869671:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7049 00000100:00080000:3.0:1788575900.869674:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575900.869677:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575900.869681:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575900.869691:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488955520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575900.869789:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488955008:192.168.202.125@tcp:250:kworker.0 00000800:00000100:2.0:1788575900.869848:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575900.869881:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575900/real 1788575900] req@ffff94d408c55f80 x1875450488955008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575900.869897:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488955008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575900.869906:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575900.869910:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575900.869914:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488955008:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575900.869921:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488955136:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575900.869943:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575900.869955:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575900/real 1788575900] req@ffff94d408c54700 x1875450488955136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575900.869966:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488955136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575900.869975:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575900.869977:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575900.869980:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488955136:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575900.869986:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488955264:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575900.870006:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575900.870018:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575900/real 1788575900] req@ffff94d408c56300 x1875450488955264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575900.870029:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488955264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575900.870043:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575900.870046:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575900.870049:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488955264:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575900.870055:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488955392:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575900.870075:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575900.870086:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575900/real 1788575900] req@ffff94d408c55880 x1875450488955392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575900.870098:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488955392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575900.870106:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575900.870109:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575900.870111:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488955392:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575900.870118:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488955520:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575900.870135:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575900.870146:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575900/real 1788575900] req@ffff94d408c55180 x1875450488955520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575900.870157:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488955520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575900.870164:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575900.870166:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575900.870170:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488955520:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575901.381333:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575901.829480:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575902.405331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575902.853367:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575903.429336:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575903.877388:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575904.453321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575904.904841:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575905.477350:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575905.926763:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575905.990124:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575905.990131:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575905.990135:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575905.990153:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7054 00000100:00080000:1.0:1788575905.990163:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575905.990170:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575905.990251:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488955648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575905.990275:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575905.990278:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575905.990280:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575905.990283:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7054 00000100:00080000:1.0:1788575905.990287:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575905.990291:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575905.990300:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575905.990311:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488955776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575905.990320:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575905.990323:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575905.990324:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575905.990327:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7054 00000100:00080000:1.0:1788575905.990330:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575905.990332:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575905.990336:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575905.990344:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488955904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575905.990351:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575905.990354:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575905.990355:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575905.990358:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7054 00000100:00080000:1.0:1788575905.990360:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575905.990363:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575905.990367:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575905.990374:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488956032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575905.990381:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575905.990384:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575905.990386:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575905.990390:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7054 00000100:00080000:1.0:1788575905.990393:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575905.990396:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575905.990400:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575905.990407:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488956160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575905.990427:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488955648:192.168.202.125@tcp:250:kworker.0 00000800:00000100:2.0:1788575905.990502:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575905.990545:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575905/real 1788575905] req@ffff94d4049bfb80 x1875450488955648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575905.990563:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488955648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575905.990573:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575905.990577:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575905.990581:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488955648:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575905.990588:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488955776:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575905.990862:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575905.990876:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575905/real 1788575905] req@ffff94d4049bc000 x1875450488955776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575905.990888:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488955776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575905.990907:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575905.990910:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575905.990913:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488955776:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575905.990919:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488955904:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575905.990939:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575905.990959:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575905/real 1788575905] req@ffff94d4049bc700 x1875450488955904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575905.990971:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488955904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575905.990979:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575905.990983:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575905.990986:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488955904:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575905.990992:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488956032:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575905.991013:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575905.991024:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575905/real 1788575905] req@ffff94d4049bed80 x1875450488956032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575905.991036:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488956032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575905.991044:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575905.991047:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575905.991051:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488956032:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575905.991057:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488956160:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575905.991075:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575905.991086:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575905/real 1788575905] req@ffff94d4049bf800 x1875450488956160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575905.991111:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488956160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575905.991119:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575905.991122:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575905.991125:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488956160:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575906.502770:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575906.951407:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575907.525555:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575907.973446:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575908.549454:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575908.997496:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575909.573459:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575910.021559:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575910.597377:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575911.050408:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575911.109367:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575911.109383:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575911.109388:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575911.109401:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7059 00000100:00080000:1.0:1788575911.109416:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575911.109425:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575911.109473:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488956288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575911.111140:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488956288:192.168.202.125@tcp:250:kworker.0 00000800:00000100:2.0:1788575911.111257:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788575911.111330:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575911.111335:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575911.111339:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575911.111345:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7059 00000100:00080000:1.0:1788575911.111350:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575911.111357:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575911.111370:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575911.111396:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488956416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575911.111581:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575911/real 1788575911] req@ffff94d4049be300 x1875450488956288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575911.111600:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488956288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575911.111610:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575911.111614:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575911.111617:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488956288:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575911.111693:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488956416:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575911.111717:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575911.111729:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575911/real 1788575911] req@ffff94d4049bf100 x1875450488956416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575911.111741:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488956416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575911.111748:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575911.111751:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575911.111754:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488956416:192.168.202.125@tcp:38:kworker.0 00000100:00080000:1.0:1788575911.111806:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575911.111810:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575911.111812:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575911.111817:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7059 00000100:00080000:1.0:1788575911.111821:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575911.111825:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575911.111830:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575911.111843:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488956544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575911.111851:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575911.111854:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575911.111855:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575911.111859:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7059 00000100:00080000:1.0:1788575911.111862:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575911.111866:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575911.111870:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575911.111878:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488956672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575911.111887:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575911.111890:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575911.111891:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575911.111894:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7059 00000100:00080000:1.0:1788575911.111897:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575911.111901:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575911.111906:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575911.111917:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488956800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575911.111978:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488956544:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575911.112001:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575911.112012:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575911/real 1788575911] req@ffff94d4049be680 x1875450488956544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575911.112022:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488956544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575911.112029:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575911.112032:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575911.112034:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488956544:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575911.112039:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488956672:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575911.112073:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575911.112083:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575911/real 1788575911] req@ffff94d4049bdc00 x1875450488956672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575911.112093:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488956672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575911.112101:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575911.112104:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575911.112107:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488956672:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575911.112112:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488956800:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575911.112129:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575911.112139:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575911/real 1788575911] req@ffff94d4049bd500 x1875450488956800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575911.112149:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488956800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575911.112156:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575911.112159:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575911.112162:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488956800:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575911.621446:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575912.069464:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575912.645335:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575913.094502:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575913.669498:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575914.117893:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575914.693301:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575915.141607:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575915.719240:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575916.165879:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575916.229389:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575916.229399:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575916.229403:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575916.229412:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7064 00000100:00080000:3.0:1788575916.229423:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575916.229431:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575916.229466:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488956928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575916.229504:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575916.229507:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575916.229509:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575916.229513:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7064 00000100:00080000:3.0:1788575916.229516:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575916.229520:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575916.229529:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575916.229541:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488957056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575916.229552:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575916.229557:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788575916.229558:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488956928:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575916.229559:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575916.229567:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7064 00000100:00080000:3.0:1788575916.229570:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575916.229573:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575916.229577:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575916.229589:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488957184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575916.229597:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575916.229601:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575916.229603:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575916.229606:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7064 00000100:00080000:3.0:1788575916.229615:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575916.229618:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575916.229623:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788575916.229626:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575916.229636:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488957312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575916.229645:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575916.229648:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575916.229649:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575916.229653:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7064 00000100:00080000:3.0:1788575916.229656:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575916.229659:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575916.229663:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575916.229665:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575916/real 1788575916] req@ffff94d408c54000 x1875450488956928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575916.229674:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488957440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575916.229692:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488956928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575916.229706:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575916.229710:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575916.229714:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488956928:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575916.229737:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488957056:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575916.229760:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575916.229772:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575916/real 1788575916] req@ffff94d408c57480 x1875450488957056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575916.229784:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488957056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575916.229797:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575916.229801:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575916.229804:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488957056:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575916.229810:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488957184:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575916.229827:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575916.229839:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575916/real 1788575916] req@ffff94d408c55180 x1875450488957184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575916.229850:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488957184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575916.229857:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575916.229860:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575916.229863:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488957184:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575916.229868:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488957312:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575916.229886:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575916.229896:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575916/real 1788575916] req@ffff94d408c55880 x1875450488957312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575916.229907:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488957312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575916.229914:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575916.229917:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575916.229920:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488957312:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575916.229925:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488957440:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575916.229941:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575916.229951:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575916/real 1788575916] req@ffff94d408c56300 x1875450488957440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575916.229960:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488957440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575916.229966:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575916.229969:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575916.229971:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488957440:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575916.741389:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575917.190549:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575917.767251:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575918.213438:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575918.789288:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575919.237498:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575919.813552:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575920.261407:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575920.837287:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575921.285500:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575921.349309:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575921.349316:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575921.349319:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575921.349327:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7069 00000100:00080000:3.0:1788575921.349335:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575921.349340:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575921.349371:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488957568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575921.349392:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575921.349395:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575921.349396:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575921.349400:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7069 00000100:00080000:3.0:1788575921.349403:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575921.349407:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575921.349416:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575921.349427:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488957696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575921.349437:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575921.349440:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575921.349441:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575921.349445:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7069 00000100:00080000:3.0:1788575921.349448:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788575921.349449:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488957568:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575921.349450:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575921.349454:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575921.349463:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488957824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575921.349471:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575921.349473:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575921.349475:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575921.349478:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7069 00000100:00080000:3.0:1788575921.349480:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575921.349484:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575921.349488:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575921.349495:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488957952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575921.349502:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788575921.349503:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575921.349505:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575921.349507:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575921.349509:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7069 00000100:00080000:3.0:1788575921.349512:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575921.349514:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575921.349517:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575921.349524:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488958080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575921.349541:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575921/real 1788575921] req@ffff94d408c54700 x1875450488957568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575921.349557:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488957568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575921.349566:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575921.349569:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575921.349573:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488957568:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575921.349590:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488957696:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575921.349607:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575921.349614:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575921/real 1788575921] req@ffff94d408c55f80 x1875450488957696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575921.349621:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488957696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575921.349625:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575921.349627:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575921.349629:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488957696:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575921.349632:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488957824:192.168.202.125@tcp:38:kworker.0 00000800:00000100:2.0:1788575921.349641:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575921.349646:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575921/real 1788575921] req@ffff94d408c57800 x1875450488957824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575921.349652:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488957824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575921.349656:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575921.349658:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575921.349660:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488957824:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575921.349662:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488957952:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575921.349671:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575921.349686:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575921/real 1788575921] req@ffff94d408c56a00 x1875450488957952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575921.349696:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488957952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575921.349703:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575921.349706:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575921.349709:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488957952:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575921.349714:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488958080:192.168.202.125@tcp:8:kworker.0 00000800:00000100:2.0:1788575921.349737:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575921.349745:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575921/real 1788575921] req@ffff94d408c54380 x1875450488958080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575921.349756:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488958080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575921.349763:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575921.349766:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:2.0:1788575921.349768:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488958080:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575921.861294:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575922.309375:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575922.885328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575923.333394:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575923.909330:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575924.357390:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575924.933300:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575925.381384:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575925.960024:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575926.405350:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575926.469321:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575926.469331:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575926.469337:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575926.469347:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7074 00000100:00080000:2.0:1788575926.469357:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575926.469363:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575926.469397:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488958208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575926.469425:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575926.469428:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575926.469430:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575926.469433:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7074 00000100:00080000:2.0:1788575926.469436:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575926.469440:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575926.469448:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575926.469459:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488958336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575926.469470:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575926.469471:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488958208:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788575926.469477:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575926.469481:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575926.469485:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7074 00000100:00080000:2.0:1788575926.469487:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575926.469490:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575926.469495:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575926.469506:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488958464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575926.469520:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575926.469524:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575926.469525:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575926.469529:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7074 00000800:00000100:0.0:1788575926.469534:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788575926.469535:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575926.469541:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575926.469547:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575926.469557:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488958592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575926.469566:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575926.469571:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575926/real 1788575926] req@ffff94d408c60a80 x1875450488958208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575926.469572:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575926.469575:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575926.469582:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7074 00000100:00080000:2.0:1788575926.469588:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575926.469592:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575926.469597:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575926.469599:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488958208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575926.469608:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488958720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575926.469636:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575926.469642:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575926.469647:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488958208:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575926.469670:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488958336:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575926.469688:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575926.469699:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575926/real 1788575926] req@ffff94d408c63b80 x1875450488958336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575926.469710:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488958336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575926.469718:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575926.469721:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575926.469724:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488958336:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575926.469729:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488958464:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575926.469743:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575926.469751:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575926/real 1788575926] req@ffff94d408c62a00 x1875450488958464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575926.469762:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488958464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575926.469769:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575926.469772:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575926.469775:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488958464:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575926.469780:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488958592:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575926.469794:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575926.469802:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575926/real 1788575926] req@ffff94d408c61f80 x1875450488958592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575926.469812:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488958592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575926.469819:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575926.469822:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575926.469825:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488958592:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575926.469830:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488958720:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575926.469858:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575926.469867:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575926/real 1788575926] req@ffff94d408c63480 x1875450488958720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575926.469877:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488958720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575926.469885:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575926.469887:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575926.469890:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488958720:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575926.981327:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575927.429383:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575928.005379:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575928.453382:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575929.029344:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575929.477434:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575930.053281:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575930.501356:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575931.077291:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575931.525400:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575931.589304:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575931.589314:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575931.589319:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575931.589329:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7079 00000100:00080000:2.0:1788575931.589338:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575931.589347:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575931.589381:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488958848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575931.589412:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575931.589416:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575931.589418:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575931.589422:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7079 00000100:00080000:2.0:1788575931.589425:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575931.589429:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575931.589437:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575931.589450:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488958976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575931.589461:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575931.589464:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575931.589466:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575931.589473:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7079 00000100:00100000:3.0:1788575931.589475:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488958848:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788575931.589477:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575931.589485:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575931.589491:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575931.589506:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488959104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575931.589516:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575931.589520:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575931.589521:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575931.589525:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7079 00000100:00080000:2.0:1788575931.589528:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575931.589531:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575931.589540:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575931.589553:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60380 x1875450488959232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575931.589562:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575931.589565:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575931.589567:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575931.589571:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7079 00000100:00080000:2.0:1788575931.589589:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575931.589594:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575931.589599:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575931.589614:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60700 x1875450488959360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575931.589697:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575931.589771:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575931/real 1788575931] req@ffff94d408c61180 x1875450488958848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575931.589792:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488958848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575931.589802:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575931.589805:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575931.589809:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488958848:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575931.589816:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488958976:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575931.589838:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575931.589850:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575931/real 1788575931] req@ffff94d408c62d80 x1875450488958976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575931.589875:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488958976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575931.589883:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575931.589886:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575931.589890:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488958976:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575931.589910:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488959104:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575931.589930:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575931.589940:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575931/real 1788575931] req@ffff94d408c60000 x1875450488959104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575931.589951:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488959104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575931.589959:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575931.589962:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575931.589965:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488959104:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575931.589971:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488959232:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575931.589988:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575931.589998:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575931/real 1788575931] req@ffff94d408c60380 x1875450488959232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575931.590009:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60380 x1875450488959232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575931.590015:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575931.590018:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575931.590022:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488959232:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575931.590027:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488959360:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575931.590042:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575931.590053:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575931/real 1788575931] req@ffff94d408c60700 x1875450488959360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575931.590063:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60700 x1875450488959360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575931.590070:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575931.590073:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575931.590076:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488959360:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575932.101295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575932.549384:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575933.125317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575933.573359:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575934.149328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575934.597404:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575935.173311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575935.621390:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575936.197313:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575936.645391:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575936.709306:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575936.709316:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575936.709321:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575936.709330:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7085 00000100:00080000:2.0:1788575936.709339:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575936.709347:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575936.709383:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62300 x1875450488959488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575936.709412:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575936.709416:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575936.709417:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575936.709421:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7085 00000100:00080000:2.0:1788575936.709424:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575936.709428:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575936.709436:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575936.709449:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488959616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575936.709460:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575936.709465:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788575936.709465:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488959488:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788575936.709467:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575936.709472:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7085 00000100:00080000:2.0:1788575936.709478:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575936.709484:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575936.709500:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575936.709514:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488959744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575936.709524:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575936.709528:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575936.709530:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575936.709534:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7085 00000100:00080000:2.0:1788575936.709538:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788575936.709539:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788575936.709547:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575936.709553:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575936.709566:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60700 x1875450488959872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575936.709575:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575936/real 1788575936] req@ffff94d408c62300 x1875450488959488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575936.709577:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575936.709596:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1788575936.709598:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62300 x1875450488959488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575936.709599:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575936.709607:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7085 00000100:00080000:2.0:1788575936.709615:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575936.709615:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575936.709621:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:2.0:1788575936.709623:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575936.709626:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488959488:192.168.202.125@tcp:250:kworker.0 00000008:00100000:2.0:1788575936.709631:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575936.709646:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60380 x1875450488960000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575936.709648:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488959616:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575936.709671:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575936.709681:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575936/real 1788575936] req@ffff94d408c62680 x1875450488959616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575936.709689:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488959616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575936.709694:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575936.709695:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575936.709697:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488959616:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575936.709700:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488959744:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788575936.709709:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575936.709714:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575936/real 1788575936] req@ffff94d408c63800 x1875450488959744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575936.709724:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488959744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575936.709729:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575936.709730:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575936.709731:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488959744:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788575936.709734:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488959872:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575936.709743:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575936.709748:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575936/real 1788575936] req@ffff94d408c60700 x1875450488959872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575936.709754:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60700 x1875450488959872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575936.709758:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575936.709759:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575936.709761:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488959872:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788575936.709769:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488960000:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788575936.709777:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788575936.709782:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575936/real 1788575936] req@ffff94d408c60380 x1875450488960000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575936.709787:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60380 x1875450488960000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575936.709796:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575936.709798:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788575936.709801:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488960000:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575937.221318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575937.671714:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575938.245325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575938.693431:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575939.269332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575939.717424:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575940.293326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575940.741324:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575941.317335:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575941.765384:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575941.829283:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575941.829293:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575941.829297:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575941.829306:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7090 00000100:00080000:3.0:1788575941.829316:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575941.829323:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575941.829360:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488960128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575941.829385:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575941.829389:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575941.829391:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575941.829395:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7090 00000100:00080000:3.0:1788575941.829398:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575941.829402:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575941.829411:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575941.829422:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488960256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575941.829430:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575941.829434:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788575941.829436:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488960128:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575941.829436:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575941.829452:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7090 00000100:00080000:3.0:1788575941.829457:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575941.829461:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575941.829465:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575941.829478:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488960384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575941.829494:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575941.829498:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1788575941.829500:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575941.829501:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575941.829508:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7090 00000100:00080000:3.0:1788575941.829512:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575941.829517:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575941.829522:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575941.829536:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488960512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575941.829540:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575941/real 1788575941] req@ffff94d408c55c00 x1875450488960128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575941.829550:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575941.829556:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575941.829559:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575941.829564:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7090 00000100:00080000:3.0:1788575941.829567:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575941.829570:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575941.829574:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488960128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788575941.829576:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575941.829587:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488960640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575941.829590:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575941.829599:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575941.829605:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488960128:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575941.829629:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488960256:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575941.829652:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575941.829664:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575941/real 1788575941] req@ffff94d408c54e00 x1875450488960256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575941.829677:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488960256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575941.829685:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575941.829688:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575941.829691:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488960256:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575941.829698:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488960384:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575941.829717:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575941.829729:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575941/real 1788575941] req@ffff94d408c57b80 x1875450488960384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575941.829741:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488960384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575941.829749:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575941.829752:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575941.829755:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488960384:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575941.829781:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488960512:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575941.829802:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575941.829812:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575941/real 1788575941] req@ffff94d408c56d80 x1875450488960512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575941.829824:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488960512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575941.829831:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575941.829834:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575941.829837:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488960512:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575941.829843:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488960640:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575941.829859:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575941.829870:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575941/real 1788575941] req@ffff94d408c56680 x1875450488960640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575941.829881:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488960640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575941.829888:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575941.829891:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575941.829894:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488960640:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575942.341322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575942.789312:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575943.365290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575943.813379:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575944.389283:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575944.837347:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575945.413322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575945.861787:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575946.437365:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575946.885819:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575946.949372:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575946.949385:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575946.949391:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575946.949404:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7095 00000100:00080000:1.0:1788575946.949416:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575946.949425:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575946.949472:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd880 x1875450488960768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575946.949501:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575946.949505:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575946.949506:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575946.949513:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7095 00000100:00080000:1.0:1788575946.949517:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575946.949521:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575946.949531:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575946.949543:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488960896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575946.949553:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575946.949559:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575946.949560:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575946.949566:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7095 00000100:00100000:0.0:1788575946.949568:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488960768:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788575946.949569:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575946.949577:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575946.949582:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575946.949595:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488961024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575946.949604:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575946.949607:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575946.949609:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575946.949612:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7095 00000100:00080000:1.0:1788575946.949616:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575946.949619:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575946.949625:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575946.949635:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488961152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575946.949643:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575946.949647:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575946.949648:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575946.949651:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7095 00000100:00080000:1.0:1788575946.949655:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575946.949658:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788575946.949664:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575946.949673:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488961280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575946.949703:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488960896:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575946.949760:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488961024:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575946.949787:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488961152:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575946.949810:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488961280:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575947.461332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575947.909515:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575948.485311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575948.933358:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575949.509463:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575949.957328:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788575950.021449:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:2.0:1788575950.021485:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788575950.021499:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788575950.021540:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788575950.021548:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788575950.021553:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788575950.021557:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:0.0:1788575950.021637:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575946/real 1788575950] req@ffff94d4049bd500 x1875450488961024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575950.021660:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488961024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575950.021670:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575950.021675:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575950.021678:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488961024:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575950.021717:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575946/real 1788575950] req@ffff94d4049bdc00 x1875450488961152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575950.021730:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488961152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575950.021738:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575950.021742:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575950.021745:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488961152:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575950.021752:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575946/real 1788575950] req@ffff94d4049be680 x1875450488961280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575950.021763:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488961280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575950.021769:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575950.021772:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575950.021775:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488961280:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575950.021781:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575946/real 1788575950] req@ffff94d4049bd880 x1875450488960768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575950.021790:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd880 x1875450488960768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575950.021797:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575950.021801:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575950.021804:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488960768:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575950.021810:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575946/real 1788575950] req@ffff94d4049bca80 x1875450488960896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575950.021822:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488960896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575950.021830:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575950.021833:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575950.021836:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488960896:192.168.202.125@tcp:38:kworker.0 00000100:00100000:2.0:1788575950.533328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575950.981363:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575951.557433:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575952.005389:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575952.069483:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575952.069494:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575952.069498:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575952.069507:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7100 00000100:00080000:2.0:1788575952.069518:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575952.069525:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575952.069560:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488961408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575952.069585:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575952.069589:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575952.069591:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575952.069595:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7100 00000100:00080000:2.0:1788575952.069599:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575952.069602:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575952.069611:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575952.069635:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488961536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575952.069640:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488961408:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788575952.069650:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575952.069659:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575952.069661:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575952.069678:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7100 00000100:00080000:2.0:1788575952.069683:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575952.069687:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575952.069692:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575952.069706:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488961664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575952.069717:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788575952.069718:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788575952.069724:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575952.069726:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575952.069731:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7100 00000100:00080000:2.0:1788575952.069736:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575952.069740:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575952.069746:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575952.069758:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575952/real 1788575952] req@ffff94d408c60000 x1875450488961408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575952.069759:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488961792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575952.069788:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575952.069797:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488961408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575952.069806:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575952.069810:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575952.069813:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7100 00000100:00080000:2.0:1788575952.069819:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575952.069823:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575952.069826:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00080000:0.0:1788575952.069832:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000008:00100000:2.0:1788575952.069833:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575952.069838:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488961408:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788575952.069852:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488961920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575952.069876:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488961536:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575952.071736:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575952.071795:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575952/real 1788575952] req@ffff94d408c62d80 x1875450488961536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575952.071819:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488961536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575952.071830:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575952.071835:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575952.071839:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488961536:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575952.071848:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488961664:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575952.071898:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575952.071909:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575952/real 1788575952] req@ffff94d408c61180 x1875450488961664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575952.071921:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488961664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575952.071933:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575952.071937:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575952.071940:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488961664:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575952.071947:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488961792:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575952.071966:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575952.071975:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575952/real 1788575952] req@ffff94d408c63480 x1875450488961792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575952.071986:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488961792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575952.071999:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575952.072002:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575952.072005:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488961792:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575952.072010:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488961920:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575952.072045:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575952.072056:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575952/real 1788575952] req@ffff94d408c61f80 x1875450488961920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575952.072087:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488961920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575952.072095:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575952.072099:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575952.072102:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488961920:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575952.581298:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575953.029374:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575953.605422:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575954.053399:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575954.629328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575955.077371:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575955.655244:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575956.101512:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575956.677600:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575957.125734:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575957.189369:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575957.189381:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575957.189385:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575957.189396:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7105 00000100:00080000:2.0:1788575957.189407:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575957.189415:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575957.189465:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488962048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575957.189507:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575957.189512:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575957.189514:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575957.189518:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7105 00000100:00080000:2.0:1788575957.189522:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575957.189526:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575957.189537:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575957.189552:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488962176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575957.189562:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575957.189566:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788575957.189572:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488962048:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788575957.189579:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575957.189589:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7105 00000100:00080000:2.0:1788575957.189594:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575957.189598:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575957.189603:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575957.189624:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488962304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575957.189635:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575957.189638:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575957.189639:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575957.189642:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7105 00000100:00080000:2.0:1788575957.189653:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575957.189656:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575957.189661:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788575957.189666:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:2.0:1788575957.189673:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488962432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575957.189680:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575957.189682:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575957.189683:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575957.189685:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7105 00000100:00080000:2.0:1788575957.189688:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575957.189690:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575957.189692:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575957.189698:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488962560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575957.189721:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575957/real 1788575957] req@ffff94d408c62a00 x1875450488962048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575957.189744:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488962048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575957.189759:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575957.189763:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575957.189767:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488962048:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575957.189808:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488962176:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575957.189833:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575957.189846:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575957/real 1788575957] req@ffff94d408c63b80 x1875450488962176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575957.189859:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488962176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575957.189866:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575957.189870:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575957.189873:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488962176:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575957.189879:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488962304:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575957.189898:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575957.189909:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575957/real 1788575957] req@ffff94d408c60a80 x1875450488962304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575957.189920:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488962304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575957.189929:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575957.189932:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575957.189935:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488962304:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575957.189942:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488962432:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575957.189963:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575957.189975:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575957/real 1788575957] req@ffff94d408c61500 x1875450488962432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575957.189987:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488962432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575957.189994:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575957.189998:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575957.190001:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488962432:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575957.190007:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488962560:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575957.190027:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575957.190038:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575957/real 1788575957] req@ffff94d408c61880 x1875450488962560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575957.190049:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488962560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575957.190057:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575957.190060:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575957.190063:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488962560:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575957.701349:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575958.149358:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575958.725550:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575959.173458:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575959.749461:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575960.197434:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575960.774114:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575961.221428:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575961.799258:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575962.245889:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575962.309333:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575962.309342:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575962.309347:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575962.309362:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7110 00000100:00080000:3.0:1788575962.309371:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575962.309378:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575962.309412:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488962688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575962.309441:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575962.309444:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575962.309446:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575962.309449:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7110 00000100:00080000:3.0:1788575962.309453:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575962.309456:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575962.309465:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575962.309481:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488962688:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575962.309481:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488962816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575962.309495:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575962.309498:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575962.309500:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575962.309504:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7110 00000100:00080000:3.0:1788575962.309508:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575962.309512:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575962.309517:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575962.309529:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488962944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575962.309535:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575962.309543:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575962.309547:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575962.309549:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575962.309554:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7110 00000100:00080000:3.0:1788575962.309560:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575962.309564:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575962.309584:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575962.309590:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575962/real 1788575962] req@ffff94d408c57100 x1875450488962688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575962.309599:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488963072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575962.309610:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488962688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575962.309611:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575962.309620:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575962.309623:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575962.309625:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575962.309629:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7110 00000100:00080000:0.0:1788575962.309631:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:3.0:1788575962.309636:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788575962.309637:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488962688:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575962.309643:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575962.309650:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575962.309660:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488963200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575962.309666:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488962816:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575962.309695:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575962.309708:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575962/real 1788575962] req@ffff94d408c54a80 x1875450488962816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575962.309718:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488962816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575962.309725:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575962.309728:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575962.309731:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488962816:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575962.309736:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488962944:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788575962.309751:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575962.309761:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575962/real 1788575962] req@ffff94d408c56680 x1875450488962944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575962.309777:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488962944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575962.309784:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575962.309787:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575962.309790:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488962944:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788575962.309795:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488963072:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575962.309814:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575962.309824:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575962/real 1788575962] req@ffff94d408c56d80 x1875450488963072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575962.309835:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488963072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575962.309843:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575962.309846:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575962.309849:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488963072:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575962.309863:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488963200:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788575962.309881:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788575962.309891:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575962/real 1788575962] req@ffff94d408c57b80 x1875450488963200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575962.309900:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488963200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575962.309913:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575962.309916:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788575962.309918:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488963200:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575962.822227:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575963.269533:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575963.845296:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575964.293494:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575964.869524:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575965.317381:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575965.893359:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575966.341380:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575966.917569:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575967.365438:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575967.430256:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575967.430269:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575967.430275:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575967.430289:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7115 00000100:00080000:3.0:1788575967.430304:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575967.430314:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575967.430360:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488963328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575967.430399:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575967.430404:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575967.430406:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575967.430412:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7115 00000100:00080000:3.0:1788575967.430417:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575967.430423:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575967.430435:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575967.430455:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488963456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575967.430468:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488963328:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575967.430477:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575967.430483:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575967.430485:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575967.430491:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7115 00000100:00080000:3.0:1788575967.430495:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788575967.430557:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575967.430597:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575967/real 1788575967] req@ffff94d408c54e00 x1875450488963328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575967.430616:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488963328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575967.430628:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575967.430632:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575967.430636:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488963328:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575967.430661:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488963456:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575967.430690:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575967.430705:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575967/real 1788575967] req@ffff94d408c55c00 x1875450488963456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575967.430722:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488963456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575967.430731:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575967.430735:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00080000:3.0:1788575967.430744:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575967.430745:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488963456:192.168.202.125@tcp:38:kworker.0 00000008:00100000:3.0:1788575967.430759:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575967.430820:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488963584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575967.430841:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575967.430845:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575967.430848:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575967.430854:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7115 00000100:00080000:3.0:1788575967.430860:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575967.430865:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575967.430871:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488963584:192.168.202.125@tcp:38:kworker.0 00000008:00100000:3.0:1788575967.430872:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575967.430897:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488963712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575967.430910:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788575967.430911:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788575967.430919:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575967.430923:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575967.430930:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7115 00000100:00100000:1.0:1788575967.430934:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575967/real 1788575967] req@ffff94d408c54380 x1875450488963584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575967.430937:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575967.430949:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575967.430957:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575967.430962:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488963584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575967.430973:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788575967.430976:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488963840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575967.430982:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575967.430993:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488963584:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575967.431028:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488963712:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575967.431057:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575967.431081:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575967/real 1788575967] req@ffff94d408c56a00 x1875450488963712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575967.431111:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488963712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575967.431119:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575967.431124:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575967.431129:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488963712:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575967.431139:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488963840:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575967.431163:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575967.431181:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575967/real 1788575967] req@ffff94d408c57800 x1875450488963840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575967.431242:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488963840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575967.431251:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575967.431255:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575967.431258:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488963840:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575967.941292:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575968.389755:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575968.965324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575969.413575:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575969.989312:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575970.437377:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575971.013321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575971.461469:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575972.037319:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575972.485373:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575972.549351:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575972.549362:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575972.549366:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575972.549376:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7120 00000100:00080000:3.0:1788575972.549388:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575972.549397:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575972.549438:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488963968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575972.549469:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575972.549503:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575972.549506:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575972.549511:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7120 00000100:00080000:3.0:1788575972.549515:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575972.549519:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575972.549529:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575972.549534:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488963968:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788575972.549546:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488964096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575972.549557:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575972.549561:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575972.549563:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575972.549566:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7120 00000100:00080000:3.0:1788575972.549570:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575972.549574:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575972.549579:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575972.549589:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488964224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575972.549597:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575972.549600:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575972.549602:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575972.549604:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7120 00000100:00080000:3.0:1788575972.549608:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575972.549610:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575972.549614:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575972.549622:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488964352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575972.549629:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575972.549631:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575972.549633:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575972.549635:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7120 00000100:00080000:3.0:1788575972.549638:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575972.549641:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575972.549646:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575972.549654:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488964480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575972.549712:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575972.549748:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575972/real 1788575972] req@ffff94d408c55f80 x1875450488963968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575972.549766:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488963968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575972.549777:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575972.549781:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575972.549785:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488963968:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575972.549809:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488964096:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575972.549830:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575972.549843:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575972/real 1788575972] req@ffff94d408c54700 x1875450488964096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575972.549855:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488964096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575972.549870:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575972.549873:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575972.549876:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488964096:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575972.549882:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488964224:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575972.549900:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575972.549911:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575972/real 1788575972] req@ffff94d408c56300 x1875450488964224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575972.549921:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488964224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575972.549929:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575972.549932:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575972.549935:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488964224:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575972.549942:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488964352:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575972.549961:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575972.549972:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575972/real 1788575972] req@ffff94d408c55880 x1875450488964352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575972.549983:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488964352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575972.549990:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575972.549993:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575972.549997:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488964352:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575972.550002:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488964480:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575972.550021:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575972.550031:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575972/real 1788575972] req@ffff94d408c55180 x1875450488964480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575972.550042:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488964480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575972.550047:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575972.550049:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575972.550051:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488964480:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575973.061319:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575973.509392:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575974.085334:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575974.533366:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575975.109306:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575975.557506:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575976.133364:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575976.581367:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575977.157313:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575977.605575:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575977.670652:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575977.670662:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575977.670666:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575977.670676:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7125 00000100:00080000:3.0:1788575977.670686:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575977.670693:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788575977.670728:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488964608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575977.670757:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575977.670761:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575977.670763:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575977.670766:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7125 00000100:00080000:3.0:1788575977.670770:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575977.670774:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575977.670782:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575977.670795:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488964736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575977.670805:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575977.670810:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788575977.670811:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488964608:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788575977.670814:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575977.670821:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7125 00000100:00080000:3.0:1788575977.670825:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575977.670829:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575977.670834:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575977.670847:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488964864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575977.670856:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575977.670860:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575977.670862:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575977.670866:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7125 00000100:00080000:3.0:1788575977.670869:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575977.670876:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:1.0:1788575977.670877:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:3.0:1788575977.670884:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575977.670898:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488964992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575977.670907:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575977.670910:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575977.670912:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575977.670918:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575977/real 1788575977] req@ffff94d408c57480 x1875450488964608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575977.670918:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7125 00000100:00080000:3.0:1788575977.670925:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575977.670932:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788575977.670939:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575977.670944:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488964608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575977.670953:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488965120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575977.670957:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575977.670966:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575977.670973:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488964608:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575977.670995:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488964736:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575977.671017:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575977.671029:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575977/real 1788575977] req@ffff94d408c54000 x1875450488964736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575977.671042:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488964736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575977.671049:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575977.671052:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575977.671056:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488964736:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575977.671061:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488964864:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575977.671080:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575977.671090:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575977/real 1788575977] req@ffff94d408c55180 x1875450488964864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575977.671101:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488964864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575977.671109:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575977.671111:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575977.671114:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488964864:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575977.671120:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488964992:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575977.671137:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575977.671146:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575977/real 1788575977] req@ffff94d408c55880 x1875450488964992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575977.671157:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488964992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575977.671165:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575977.671168:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575977.671171:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488964992:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575977.671177:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488965120:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575977.671192:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575977.671231:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575977/real 1788575977] req@ffff94d408c56300 x1875450488965120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575977.671244:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488965120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575977.671251:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575977.671254:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575977.671258:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488965120:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788575978.181336:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575978.629403:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575979.205331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575979.653314:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575980.229309:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575980.677344:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575981.253279:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575981.701475:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575982.277924:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575982.725381:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575982.789383:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575982.789395:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575982.789399:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575982.789409:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7131 00000100:00080000:0.0:1788575982.789422:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575982.789431:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575982.789479:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f800 x1875450488965248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575982.789508:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575982.789511:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575982.789513:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575982.789515:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7131 00000100:00080000:0.0:1788575982.789518:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575982.789521:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575982.789531:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575982.789540:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f100 x1875450488965376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575982.789547:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575982.789550:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575982.789552:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575982.789555:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7131 00000100:00080000:0.0:1788575982.789560:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575982.789562:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575982.789565:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575982.789577:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488965504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575982.789586:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575982.789589:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575982.789592:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575982.789601:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488965248:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788575982.789602:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7131 00000100:00080000:0.0:1788575982.789607:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575982.789616:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575982.789632:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575982.789652:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488965632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575982.789660:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575982.789664:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575982.789667:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575982.789672:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7131 00000100:00080000:0.0:1788575982.789679:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575982.789693:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575982.789699:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575982.789715:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488965760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575982.789719:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575982.789776:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575982/real 1788575982] req@ffff94d408d3f800 x1875450488965248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575982.789805:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f800 x1875450488965248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575982.789813:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575982.789816:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575982.789818:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488965248:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575982.789825:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488965376:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575982.789845:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575982.789853:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575982/real 1788575982] req@ffff94d408d3f100 x1875450488965376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575982.789862:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f100 x1875450488965376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575982.789869:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575982.789872:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575982.789875:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488965376:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575982.789900:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488965504:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575982.789925:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575982.789937:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575982/real 1788575982] req@ffff94d408d3d500 x1875450488965504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575982.789948:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488965504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575982.789956:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575982.789959:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575982.789962:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488965504:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575982.789979:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488965632:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575982.789997:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575982.790005:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575982/real 1788575982] req@ffff94d408d3c000 x1875450488965632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575982.790013:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488965632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575982.790017:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575982.790019:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575982.790021:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488965632:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575982.790025:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488965760:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575982.790050:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575982.790060:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575982/real 1788575982] req@ffff94d408d3e680 x1875450488965760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575982.790071:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488965760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575982.790079:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575982.790082:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575982.790086:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488965760:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575983.301406:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575983.749407:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575984.327845:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575984.773359:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575985.349324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575985.797379:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575986.373914:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575986.821403:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575987.397308:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575987.845355:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575987.909355:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575987.909366:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575987.909371:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575987.909381:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7136 00000100:00080000:2.0:1788575987.909393:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575987.909400:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788575987.909440:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488965888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575987.909465:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575987.909467:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575987.909468:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575987.909472:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7136 00000100:00080000:2.0:1788575987.909475:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575987.909479:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575987.909489:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575987.909502:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60e00 x1875450488966016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575987.909513:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575987.909516:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488965888:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788575987.909518:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575987.909523:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575987.909528:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7136 00000100:00080000:2.0:1788575987.909531:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575987.909534:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575987.909539:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575987.909548:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63100 x1875450488966144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575987.909559:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575987.909562:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575987.909563:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575987.909565:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7136 00000100:00080000:2.0:1788575987.909568:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575987.909570:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575987.909572:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575987.909583:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488966272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575987.909592:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575987.909595:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575987.909596:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575987.909600:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7136 00000800:00000100:1.0:1788575987.909605:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788575987.909607:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575987.909613:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788575987.909619:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575987.909630:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488966400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575987.909653:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575987/real 1788575987] req@ffff94d408c61c00 x1875450488965888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575987.909673:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488965888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575987.909691:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575987.909695:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575987.909699:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488965888:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575987.909724:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488966016:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575987.909750:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575987.909760:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575987/real 1788575987] req@ffff94d408c60e00 x1875450488966016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575987.909769:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60e00 x1875450488966016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575987.909774:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575987.909776:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575987.909778:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488966016:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575987.909782:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488966144:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575987.909797:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575987.909807:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575987/real 1788575987] req@ffff94d408c63100 x1875450488966144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575987.909818:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63100 x1875450488966144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575987.909826:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575987.909830:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575987.909832:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488966144:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575987.909836:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488966272:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575987.909875:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575987.909885:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575987/real 1788575987] req@ffff94d408c61880 x1875450488966272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575987.909895:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488966272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575987.909902:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575987.909905:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575987.909908:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488966272:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575987.909914:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488966400:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575987.909931:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575987.909941:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575987/real 1788575987] req@ffff94d408c61500 x1875450488966400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575987.909952:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488966400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575987.909957:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575987.909959:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575987.909961:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488966400:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575988.421357:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575988.869428:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575989.445357:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575989.893428:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575990.469370:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575990.917398:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575991.493347:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575991.941443:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575992.517363:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575992.965567:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575993.029896:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575993.029912:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575993.029918:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575993.029932:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7141 00000100:00080000:0.0:1788575993.029945:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575993.029954:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575993.030029:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488966528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575993.030074:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575993.030079:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575993.030081:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575993.030085:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7141 00000100:00080000:0.0:1788575993.030089:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575993.030093:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575993.030104:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575993.030116:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488966656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575993.030138:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488966528:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788575993.030139:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575993.030148:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575993.030151:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575993.030156:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7141 00000100:00080000:0.0:1788575993.030160:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575993.030165:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575993.030170:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575993.030196:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488966784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575993.030257:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575993.030261:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575993.030263:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575993.030266:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7141 00000100:00080000:0.0:1788575993.030271:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788575993.030276:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788575993.030280:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575993.030286:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575993.030299:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488966912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575993.030317:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575993.030320:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575993.030322:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575993.030326:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7141 00000100:00080000:0.0:1788575993.030330:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575993.030332:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575993/real 1788575993] req@ffff94d408d3ea00 x1875450488966528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575993.030337:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575993.030345:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575993.030358:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488967040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575993.030362:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488966528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575993.030383:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575993.030388:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575993.030393:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488966528:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575993.030424:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488966656:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575993.030447:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575993.030460:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575993/real 1788575993] req@ffff94d408d3ca80 x1875450488966656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575993.030472:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488966656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575993.030480:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575993.030483:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575993.030486:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488966656:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575993.030491:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488966784:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575993.030509:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575993.030519:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575993/real 1788575993] req@ffff94d408d3d180 x1875450488966784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575993.030530:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488966784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575993.030543:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575993.030546:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575993.030549:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488966784:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575993.030554:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488966912:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575993.030573:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575993.030594:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575993/real 1788575993] req@ffff94d408d3c700 x1875450488966912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575993.030609:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488966912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575993.030617:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575993.030621:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575993.030624:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488966912:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575993.030629:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488967040:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575993.030649:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575993.030660:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575993/real 1788575993] req@ffff94d408d3df80 x1875450488967040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575993.030670:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488967040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575993.030677:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575993.030680:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575993.030683:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488967040:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575993.541370:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575993.989438:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575994.565320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575995.013361:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575995.589334:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575996.037368:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575996.613323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575997.061346:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575997.637294:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788575998.085374:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575998.149321:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575998.149328:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575998.149332:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575998.149340:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7146 00000100:00080000:0.0:1788575998.149348:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575998.149355:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788575998.149388:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488967168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575998.149404:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575998.149407:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575998.149408:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575998.149411:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7146 00000100:00080000:0.0:1788575998.149413:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575998.149415:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575998.149426:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575998.149439:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488967168:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788575998.149443:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488967296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575998.149455:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575998.149459:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575998.149460:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575998.149464:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7146 00000100:00080000:0.0:1788575998.149468:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575998.149472:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575998.149476:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575998.149486:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ed80 x1875450488967424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575998.149493:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575998.149496:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1788575998.149498:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788575998.149499:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575998.149509:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7146 00000100:00080000:0.0:1788575998.149512:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575998.149516:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788575998.149519:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575998.149526:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488967552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575998.149538:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575998.149541:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575998/real 1788575998] req@ffff94d408d3ce00 x1875450488967168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575998.149544:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575998.149548:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575998.149557:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7146 00000100:00080000:0.0:1788575998.149563:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575998.149567:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788575998.149571:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488967168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788575998.149573:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788575998.149587:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788575998.149591:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488967680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575998.149595:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575998.149602:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488967168:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788575998.149636:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488967296:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575998.149660:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575998.149669:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575998/real 1788575998] req@ffff94d408d3f480 x1875450488967296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575998.149677:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488967296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575998.149682:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575998.149684:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575998.149687:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488967296:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575998.149693:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488967424:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788575998.149709:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575998.149720:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575998/real 1788575998] req@ffff94d408d3ed80 x1875450488967424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575998.149732:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ed80 x1875450488967424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575998.149738:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575998.149740:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575998.149743:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488967424:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788575998.149748:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488967552:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575998.149768:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575998.149779:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575998/real 1788575998] req@ffff94d408d3c380 x1875450488967552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575998.149791:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488967552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575998.149798:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575998.149801:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575998.149804:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488967552:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788575998.149809:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488967680:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788575998.149835:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788575998.149846:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575998/real 1788575998] req@ffff94d408d3d880 x1875450488967680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575998.149857:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488967680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575998.149865:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575998.149868:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788575998.149871:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488967680:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788575998.661289:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788575999.109343:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575999.685299:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576000.133408:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576000.709388:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788576001.160303:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576001.733304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576002.181388:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576002.757345:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576003.205451:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788576003.269351:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576003.269361:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576003.269366:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576003.269389:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7151 00000100:00080000:2.0:1788576003.269399:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576003.269406:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788576003.269438:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488967808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576003.269461:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576003.269464:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576003.269465:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576003.269468:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7151 00000100:00080000:2.0:1788576003.269470:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576003.269472:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576003.269480:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576003.269488:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488967936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576003.269498:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576003.269500:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788576003.269502:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576003.269504:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7151 00000100:00080000:2.0:1788576003.269506:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576003.269508:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576003.269510:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576003.269516:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488968064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576003.269522:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576003.269524:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576003.269525:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576003.269528:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7151 00000100:00080000:2.0:1788576003.269529:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576003.269531:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576003.269533:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576003.269540:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488968192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576003.269544:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576003.269546:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788576003.269547:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576003.269549:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7151 00000100:00080000:2.0:1788576003.269551:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576003.269553:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576003.269556:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576003.269560:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488968320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576003.269675:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488967808:192.168.202.125@tcp:250:kworker.0 00000800:00000100:0.0:1788576003.269737:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576003.269767:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576003/real 1788576003] req@ffff94d408c60a80 x1875450488967808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576003.269785:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488967808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576003.269795:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576003.269851:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576003.269855:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488967808:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788576003.269863:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488967936:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576003.269895:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576003.269909:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576003/real 1788576003] req@ffff94d408c63b80 x1875450488967936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576003.269922:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488967936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576003.269931:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576003.269934:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576003.269937:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488967936:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576003.269944:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488968064:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576003.269964:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576003.269975:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576003/real 1788576003] req@ffff94d408c62a00 x1875450488968064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576003.269987:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488968064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576003.269995:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576003.269998:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576003.270001:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488968064:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576003.270007:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488968192:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576003.270043:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576003.270056:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576003/real 1788576003] req@ffff94d408c61f80 x1875450488968192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576003.270067:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488968192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576003.270074:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576003.270077:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576003.270081:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488968192:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576003.270086:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488968320:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576003.270105:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576003.270115:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576003/real 1788576003] req@ffff94d408c63480 x1875450488968320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576003.270127:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488968320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576003.270134:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576003.270137:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576003.270140:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488968320:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788576003.781279:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788576004.231226:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576004.805333:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576005.253375:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576005.829982:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576006.277382:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576006.853274:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788576007.301343:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576007.877323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576008.325429:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788576008.389382:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576008.389397:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788576008.389403:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576008.389418:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7156 00000100:00080000:3.0:1788576008.389434:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576008.389442:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788576008.389513:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488968448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576008.389538:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576008.389542:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576008.389547:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576008.389551:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7156 00000100:00080000:3.0:1788576008.389555:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576008.389559:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576008.389572:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576008.389589:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488968576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576008.389596:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488968448:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788576008.389599:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576008.389602:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788576008.389603:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576008.389607:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7156 00000100:00080000:3.0:1788576008.389610:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576008.389614:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576008.389618:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576008.389627:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488968704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576008.389634:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576008.389637:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576008.389638:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576008.389641:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7156 00000100:00080000:3.0:1788576008.389644:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576008.389648:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576008.389652:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576008.389659:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488968832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576008.389666:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576008.389669:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788576008.389671:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788576008.389673:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788576008.389674:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7156 00000100:00080000:3.0:1788576008.389677:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576008.389680:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576008.389685:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576008.389692:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488968960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576008.389716:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576008/real 1788576008] req@ffff94d408c54700 x1875450488968448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576008.389735:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488968448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576008.389746:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576008.389749:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576008.389752:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488968448:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788576008.389777:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488968576:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576008.389795:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576008.389803:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576008/real 1788576008] req@ffff94d408c55f80 x1875450488968576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576008.389811:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488968576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576008.389816:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576008.389818:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576008.389820:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488968576:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576008.389823:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488968704:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576008.389832:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576008.389837:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576008/real 1788576008] req@ffff94d408c57800 x1875450488968704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576008.389843:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488968704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576008.389848:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576008.389850:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576008.389851:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488968704:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576008.389854:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488968832:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576008.389865:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576008.389870:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576008/real 1788576008] req@ffff94d408c56a00 x1875450488968832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576008.389878:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488968832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576008.389882:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576008.389883:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576008.389885:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488968832:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576008.389888:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488968960:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576008.389896:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576008.389901:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576008/real 1788576008] req@ffff94d408c54380 x1875450488968960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576008.389907:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488968960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576008.389912:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576008.389913:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576008.389915:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488968960:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788576008.901373:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576009.349554:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576009.925589:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788576010.373821:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576010.949384:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576011.400631:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576011.973347:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576012.421423:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576012.997320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788576013.445629:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788576013.509388:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576013.509399:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576013.509404:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576013.509414:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7161 00000100:00080000:2.0:1788576013.509424:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576013.509431:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788576013.509467:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488969088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576013.509496:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576013.509500:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576013.509501:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576013.509507:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7161 00000100:00080000:2.0:1788576013.509519:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576013.509523:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576013.509533:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576013.509547:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488969216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576013.509557:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576013.509561:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788576013.509564:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788576013.509572:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488969088:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788576013.509572:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7161 00000100:00080000:2.0:1788576013.509582:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576013.509589:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576013.509594:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576013.509609:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488969344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576013.509644:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576013.509648:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576013.509650:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576013.509655:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7161 00000100:00080000:2.0:1788576013.509659:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576013.509663:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576013.509669:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576013.509682:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60380 x1875450488969472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576013.509692:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576013.509695:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788576013.509698:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576013.509702:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7161 00000100:00080000:2.0:1788576013.509708:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576013.509712:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576013.509718:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576013.509732:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60700 x1875450488969600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576013.509738:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488969216:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576013.509788:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488969344:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576013.509822:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488969472:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576013.509849:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488969600:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788576014.021290:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576014.469433:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576015.045318:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576015.496023:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576016.069326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788576016.517905:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788576016.581541:0:2392:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:1.0:1788576016.581573:0:2392:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788576016.581586:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788576016.581619:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788576016.581625:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788576016.581630:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788576016.581636:0:2392:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:0.0:1788576016.581672:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576013/real 1788576016] req@ffff94d408c60700 x1875450488969600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576016.581697:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60700 x1875450488969600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576016.581720:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576016.581725:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576016.581729:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488969600:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576016.581736:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576013/real 1788576016] req@ffff94d408c62d80 x1875450488969216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576016.581748:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488969216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576016.581754:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576016.581757:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576016.581761:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488969216:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576016.581767:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576013/real 1788576016] req@ffff94d408c60000 x1875450488969344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576016.581784:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488969344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576016.581793:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576016.581797:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576016.581800:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488969344:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576016.581806:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576013/real 1788576016] req@ffff94d408c60380 x1875450488969472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576016.581816:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60380 x1875450488969472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576016.581822:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576016.581825:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576016.581828:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488969472:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576016.581834:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576013/real 1788576016] req@ffff94d408c61180 x1875450488969088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576016.581846:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488969088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576016.581853:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576016.581867:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576016.581870:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488969088:192.168.202.125@tcp:250:kworker.0 00000100:00100000:2.0:1788576017.093285:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576017.541355:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576018.117317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576018.565376:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788576018.629391:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576018.629402:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788576018.629406:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576018.629416:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7166 00000100:00080000:3.0:1788576018.629426:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576018.629433:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788576018.629465:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488969728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576018.629545:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488969728:192.168.202.125@tcp:250:kworker.0 00000800:00000100:0.0:1788576018.629603:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788576018.629636:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576018.629640:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576018.629642:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576018.629646:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7166 00000100:00080000:3.0:1788576018.629650:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576018.629654:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576018.629664:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576018.629679:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488969856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576018.629743:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576018/real 1788576018] req@ffff94d408c55c00 x1875450488969728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576018.629761:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488969728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576018.629796:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576018.629801:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576018.629805:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488969728:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788576018.629831:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488969856:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576018.629854:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576018.629867:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576018/real 1788576018] req@ffff94d408c54e00 x1875450488969856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576018.629879:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488969856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576018.629887:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576018.629889:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576018.629893:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488969856:192.168.202.125@tcp:38:kworker.0 00000100:00080000:3.0:1788576018.629990:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576018.629995:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788576018.629997:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576018.630002:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7166 00000100:00080000:3.0:1788576018.630006:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576018.630010:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576018.630015:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576018.630028:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488969984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576018.630084:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488969984:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576018.630108:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788576018.630193:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576018.630197:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576018.630199:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576018.630224:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7166 00000100:00080000:3.0:1788576018.630229:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576018.630234:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576018.630240:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576018.630252:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488970112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576018.630818:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576018/real 1788576018] req@ffff94d408c57b80 x1875450488969984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576018.630834:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488969984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576018.630848:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576018.630851:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576018.630853:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488969984:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576018.630878:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488970112:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576018.630906:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576018.630918:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576018/real 1788576018] req@ffff94d408c56d80 x1875450488970112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576018.630930:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488970112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576018.630937:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576018.630940:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576018.630943:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488970112:192.168.202.125@tcp:8:kworker.0 00000100:00080000:3.0:1788576018.631599:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576018.631605:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788576018.631608:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576018.631613:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7166 00000100:00080000:3.0:1788576018.631619:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576018.631625:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576018.631632:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576018.631652:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488970240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576018.631699:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488970240:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576018.631730:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576018.631747:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576018/real 1788576018] req@ffff94d408c56680 x1875450488970240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576018.631780:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488970240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576018.631799:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576018.631803:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576018.631808:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488970240:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788576019.141365:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788576019.589441:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576020.165899:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576020.614790:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576021.189900:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576021.637372:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576022.213343:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788576022.662488:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576023.237580:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576023.685394:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788576023.749359:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576023.749370:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788576023.749374:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576023.749387:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7172 00000100:00080000:3.0:1788576023.749401:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576023.749424:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788576023.749472:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488970368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576023.749518:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576023.749523:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576023.749524:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576023.749528:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7172 00000100:00080000:3.0:1788576023.749531:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576023.749535:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576023.749545:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576023.749561:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488970496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576023.749573:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576023.749576:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788576023.749578:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576023.749584:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7172 00000100:00080000:3.0:1788576023.749588:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788576023.749596:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488970368:192.168.202.125@tcp:250:kworker.0 00000100:00080000:3.0:1788576023.749596:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576023.749601:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576023.749621:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488970624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576023.749631:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576023.749635:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576023.749636:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576023.749641:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7172 00000100:00080000:3.0:1788576023.749644:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576023.749649:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576023.749655:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576023.749668:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488970752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576023.749679:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576023.749682:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788576023.749683:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576023.749686:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7172 00000800:00000100:0.0:1788576023.749695:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788576023.749697:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576023.749702:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576023.749709:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576023.749719:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488970880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576023.749748:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576023/real 1788576023] req@ffff94d408c54a80 x1875450488970368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576023.749770:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488970368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576023.749781:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576023.749785:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576023.749789:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488970368:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788576023.749797:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488970496:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576023.749822:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576023.749835:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576023/real 1788576023] req@ffff94d408c57100 x1875450488970496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576023.749848:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488970496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576023.749855:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576023.749858:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576023.749862:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488970496:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576023.749886:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488970624:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576023.749903:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576023.749910:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576023/real 1788576023] req@ffff94d408c56680 x1875450488970624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576023.749917:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488970624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576023.749937:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576023.749940:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576023.749943:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488970624:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576023.749949:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488970752:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576023.749969:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576023.749979:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576023/real 1788576023] req@ffff94d408c56d80 x1875450488970752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576023.749987:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488970752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576023.749991:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576023.749994:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576023.749995:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488970752:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576023.749999:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488970880:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576023.750010:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576023.750016:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576023/real 1788576023] req@ffff94d408c57b80 x1875450488970880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576023.750023:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488970880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576023.750027:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576023.750029:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576023.750030:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488970880:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788576024.261329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576024.709576:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576025.285429:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788576025.733344:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576026.311252:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576026.757498:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576027.333295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576027.781398:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576028.357343:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788576028.806287:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788576028.869299:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576028.869308:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788576028.869312:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576028.869323:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7177 00000100:00080000:3.0:1788576028.869333:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576028.869342:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788576028.869375:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488971008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576028.869459:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576028.869463:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576028.869465:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576028.869469:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7177 00000100:00080000:3.0:1788576028.869472:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576028.869476:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576028.869484:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576028.869495:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488971136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576028.869508:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576028.869511:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788576028.869512:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576028.869515:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7177 00000100:00080000:3.0:1788576028.869517:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576028.869520:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576028.869524:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576028.869532:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488971264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576028.869540:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576028.869554:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576028.869556:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576028.869559:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7177 00000100:00080000:3.0:1788576028.869562:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576028.869565:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576028.869575:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576028.869584:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488971392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576028.869592:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576028.869594:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788576028.869596:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576028.869599:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7177 00000100:00080000:3.0:1788576028.869602:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576028.869605:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576028.869609:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576028.869616:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488971520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576028.869919:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488971008:192.168.202.125@tcp:250:kworker.0 00000800:00000100:0.0:1788576028.869976:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576028.870009:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576028/real 1788576028] req@ffff94d408c54e00 x1875450488971008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576028.870026:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488971008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576028.870036:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576028.870039:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576028.870043:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488971008:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788576028.870049:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488971136:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576028.870071:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576028.870094:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576028/real 1788576028] req@ffff94d408c55c00 x1875450488971136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576028.870112:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488971136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576028.870120:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576028.870123:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576028.870126:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488971136:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576028.870132:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488971264:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576028.870329:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576028.870345:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576028/real 1788576028] req@ffff94d408c54380 x1875450488971264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576028.870359:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488971264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576028.870367:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576028.870370:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576028.870372:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488971264:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576028.870379:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488971392:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576028.870401:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576028.870411:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576028/real 1788576028] req@ffff94d408c56a00 x1875450488971392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576028.870422:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488971392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576028.870435:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576028.870438:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576028.870441:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488971392:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576028.870447:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488971520:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576028.870466:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576028.870476:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576028/real 1788576028] req@ffff94d408c57800 x1875450488971520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576028.870488:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488971520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576028.870495:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576028.870498:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576028.870500:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488971520:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788576029.381314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576029.829885:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576030.405314:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576030.853541:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576031.429323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788576031.878989:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576032.453707:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576032.901382:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576033.477666:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576033.925380:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788576033.989374:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576033.989390:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576033.989394:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576033.989403:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7182 00000100:00080000:2.0:1788576033.989414:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576033.989421:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788576033.989456:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488971648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576033.989483:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576033.989486:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576033.989488:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576033.989491:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7182 00000100:00080000:2.0:1788576033.989494:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576033.989497:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576033.989506:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576033.989515:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488971776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576033.989525:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576033.989528:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788576033.989529:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576033.989532:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7182 00000100:00080000:2.0:1788576033.989534:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576033.989537:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576033.989542:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576033.989550:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62300 x1875450488971904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576033.989558:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576033.989561:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576033.989563:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576033.989566:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7182 00000100:00080000:2.0:1788576033.989568:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576033.989571:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576033.989575:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576033.989583:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488972032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576033.989591:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576033.989593:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788576033.989595:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576033.989598:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7182 00000100:00080000:2.0:1788576033.989600:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576033.989603:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576033.989607:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576033.989614:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60380 x1875450488972160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576033.990256:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488971648:192.168.202.125@tcp:250:kworker.0 00000800:00000100:0.0:1788576033.990311:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576033.990341:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576033/real 1788576033] req@ffff94d408c63800 x1875450488971648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576033.990357:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488971648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576033.990379:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576033.990382:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576033.990386:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488971648:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788576033.990393:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488971776:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576033.990411:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576033.990420:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576033/real 1788576033] req@ffff94d408c62680 x1875450488971776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576033.990431:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488971776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576033.990439:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576033.990441:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576033.990445:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488971776:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576033.990449:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488971904:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576033.990462:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576033.990469:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576033/real 1788576033] req@ffff94d408c62300 x1875450488971904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576033.990479:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62300 x1875450488971904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576033.990487:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576033.990489:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576033.990492:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488971904:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576033.990497:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488972032:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576033.990510:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576033.990518:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576033/real 1788576033] req@ffff94d408c61180 x1875450488972032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576033.990527:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488972032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576033.990534:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576033.990537:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576033.990540:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488972032:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576033.990544:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488972160:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576033.990558:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576033.990566:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576033/real 1788576033] req@ffff94d408c60380 x1875450488972160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576033.990576:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60380 x1875450488972160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576033.990583:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576033.990587:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576033.990589:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488972160:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788576034.501320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788576034.949391:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576035.525327:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576035.973407:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576036.549332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576036.997487:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576037.573508:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788576038.021502:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576038.597326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576039.045399:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788576039.109301:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576039.109313:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576039.109318:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576039.109328:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7187 00000100:00080000:2.0:1788576039.109339:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576039.109348:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788576039.109383:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488972288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576039.109439:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576039.109443:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576039.109445:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576039.109449:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7187 00000100:00080000:2.0:1788576039.109453:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576039.109457:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576039.109467:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576039.109483:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488972416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576039.109494:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788576039.109497:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488972288:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788576039.109499:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788576039.109503:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576039.109510:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7187 00000100:00080000:2.0:1788576039.109513:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576039.109517:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576039.109522:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576039.109548:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60700 x1875450488972544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576039.109558:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576039.109561:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576039.109563:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576039.109566:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7187 00000800:00000100:3.0:1788576039.109571:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788576039.109573:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576039.109580:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576039.109585:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576039.109598:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488972672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576039.109603:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576039/real 1788576039] req@ffff94d408c60000 x1875450488972288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576039.109612:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576039.109617:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788576039.109619:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576039.109625:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7187 00000100:00080000:2.0:1788576039.109628:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576039.109632:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576039.109637:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576039.109648:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488972800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576039.109679:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488972288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576039.109829:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576039.109834:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788576039.109839:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488972288:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788576039.109861:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488972416:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788576039.109886:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788576039.109900:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576039/real 1788576039] req@ffff94d408c62d80 x1875450488972416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576039.109913:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488972416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576039.109927:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576039.109930:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788576039.109933:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488972416:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788576039.109940:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488972544:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788576039.109961:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788576039.109973:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576039/real 1788576039] req@ffff94d408c60700 x1875450488972544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576039.109999:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60700 x1875450488972544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576039.110006:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576039.110010:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788576039.110013:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488972544:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788576039.110019:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488972672:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788576039.110039:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788576039.110049:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576039/real 1788576039] req@ffff94d408c63480 x1875450488972672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576039.110060:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488972672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576039.110068:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576039.110071:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788576039.110074:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488972672:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788576039.110080:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488972800:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788576039.110098:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788576039.110108:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576039/real 1788576039] req@ffff94d408c61f80 x1875450488972800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576039.110118:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488972800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576039.110125:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576039.110129:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788576039.110133:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488972800:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788576039.621302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576040.069430:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576040.645321:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788576041.093574:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576041.669302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576042.118606:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576042.693302:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576043.141487:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576043.717299:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788576044.165346:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788576044.229353:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576044.229364:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576044.229368:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576044.229379:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7192 00000100:00080000:0.0:1788576044.229390:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576044.229398:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788576044.229448:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488972928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576044.229478:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576044.229482:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576044.229484:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576044.229488:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7192 00000100:00080000:0.0:1788576044.229491:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576044.229495:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576044.229505:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576044.229518:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488973056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576044.229530:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788576044.229532:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488972928:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788576044.229536:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788576044.229540:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576044.229545:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7192 00000100:00080000:0.0:1788576044.229547:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576044.229550:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576044.229555:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576044.229569:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488973184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576044.229579:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576044.229582:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576044.229583:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576044.229587:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7192 00000100:00080000:0.0:1788576044.229591:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576044.229597:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:3.0:1788576044.229598:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:0.0:1788576044.229606:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576044.229617:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488973312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576044.229623:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576044.229625:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788576044.229627:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576044.229631:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7192 00000100:00080000:0.0:1788576044.229634:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788576044.229637:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576044/real 1788576044] req@ffff94d408d3e300 x1875450488972928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576044.229641:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576044.229649:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576044.229662:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ed80 x1875450488973440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576044.229664:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488972928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576044.229688:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576044.229692:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788576044.229697:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488972928:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788576044.229719:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488973056:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788576044.229741:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788576044.229753:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576044/real 1788576044] req@ffff94d408d3dc00 x1875450488973056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576044.229766:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488973056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576044.229775:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576044.229778:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788576044.229782:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488973056:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788576044.229788:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488973184:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788576044.229807:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788576044.229818:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576044/real 1788576044] req@ffff94d408d3d880 x1875450488973184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576044.229830:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488973184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576044.229855:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576044.229859:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788576044.229863:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488973184:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788576044.229869:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488973312:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788576044.229888:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788576044.229899:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576044/real 1788576044] req@ffff94d408d3c380 x1875450488973312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576044.229916:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488973312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576044.229923:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576044.229927:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788576044.229930:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488973312:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788576044.229935:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488973440:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788576044.229952:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788576044.229962:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576044/real 1788576044] req@ffff94d408d3ed80 x1875450488973440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576044.229971:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ed80 x1875450488973440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576044.229978:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576044.229981:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788576044.229984:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488973440:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788576044.741329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576045.189408:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576045.765332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576046.213475:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576046.789325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788576047.237670:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576047.813344:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576048.261373:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576048.837304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576049.286130:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788576049.349421:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576049.349444:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576049.349449:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576049.349458:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7197 00000100:00080000:2.0:1788576049.349469:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576049.349477:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788576049.349527:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488973568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576049.349555:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576049.349559:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576049.349561:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576049.349565:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7197 00000100:00080000:2.0:1788576049.349569:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576049.349572:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576049.349581:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576049.349593:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488973696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576049.349609:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576049.349613:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788576049.349615:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576049.349618:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7197 00000100:00080000:2.0:1788576049.349621:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576049.349624:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576049.349629:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576049.349653:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488973824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576049.349661:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576049.349665:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576049.349666:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576049.349670:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7197 00000100:00080000:2.0:1788576049.349674:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576049.349677:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576049.349682:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576049.349693:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488973952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576049.349702:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576049.349705:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788576049.349706:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576049.349719:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7197 00000100:00080000:2.0:1788576049.349723:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576049.349726:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576049.349730:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576049.349740:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488974080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576049.350339:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488973568:192.168.202.125@tcp:250:kworker.0 00000800:00000100:3.0:1788576049.350446:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788576049.350498:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576049/real 1788576049] req@ffff94d408c62a00 x1875450488973568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576049.350524:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488973568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576049.350537:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576049.350541:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788576049.350546:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488973568:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788576049.350555:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488973696:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788576049.350584:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788576049.350597:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576049/real 1788576049] req@ffff94d408c63b80 x1875450488973696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576049.350608:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488973696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576049.350622:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576049.350625:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788576049.350628:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488973696:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788576049.350635:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488973824:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788576049.350655:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788576049.350667:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576049/real 1788576049] req@ffff94d408c60a80 x1875450488973824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576049.350678:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488973824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576049.350686:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576049.350689:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788576049.350692:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488973824:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788576049.350699:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488973952:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788576049.350720:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788576049.350731:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576049/real 1788576049] req@ffff94d408c61500 x1875450488973952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576049.350742:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488973952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576049.350750:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576049.350754:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788576049.350757:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488973952:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788576049.350763:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488974080:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788576049.350784:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788576049.350795:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576049/real 1788576049] req@ffff94d408c61880 x1875450488974080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576049.350805:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488974080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576049.350812:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576049.350815:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788576049.350817:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488974080:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788576049.861351:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788576050.309679:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576050.885310:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576051.333420:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576051.909328:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576052.357354:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576052.933313:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788576053.381398:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576053.957316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576054.405357:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788576054.469338:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576054.469349:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576054.469353:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576054.469363:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7202 00000100:00080000:0.0:1788576054.469374:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576054.469381:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788576054.469417:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488974208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576054.469456:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576054.469460:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576054.469462:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576054.469466:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7202 00000100:00080000:0.0:1788576054.469470:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576054.469474:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576054.469483:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576054.469497:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488974336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576054.469500:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488974208:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788576054.469519:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576054.469523:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788576054.469525:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576054.469530:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7202 00000100:00080000:0.0:1788576054.469534:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576054.469538:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576054.469543:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576054.469554:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488974464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576054.469564:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576054.469567:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576054.469569:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576054.469574:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7202 00000100:00080000:0.0:1788576054.469577:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788576054.469580:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788576054.469583:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576054.469589:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576054.469600:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488974592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576054.469610:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576054.469613:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788576054.469615:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576054.469618:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7202 00000100:00080000:0.0:1788576054.469622:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788576054.469627:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576054/real 1788576054] req@ffff94d408d3f480 x1875450488974208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576054.469628:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576054.469636:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576054.469651:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488974720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576054.469660:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488974208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576054.469673:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576054.469678:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788576054.469683:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488974208:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788576054.469706:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488974336:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788576054.469728:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788576054.469752:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576054/real 1788576054] req@ffff94d408d3ce00 x1875450488974336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576054.469764:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488974336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576054.469772:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576054.469775:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788576054.469779:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488974336:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788576054.469785:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488974464:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788576054.469804:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788576054.469816:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576054/real 1788576054] req@ffff94d408d3df80 x1875450488974464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576054.469832:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488974464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576054.469841:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576054.469844:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788576054.469846:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488974464:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788576054.469852:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488974592:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788576054.469871:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788576054.469881:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576054/real 1788576054] req@ffff94d408d3c700 x1875450488974592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576054.469894:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488974592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576054.469901:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576054.469904:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788576054.469907:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488974592:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788576054.469921:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488974720:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788576054.469939:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788576054.469949:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576054/real 1788576054] req@ffff94d408d3d180 x1875450488974720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576054.469969:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488974720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576054.469982:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576054.469985:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788576054.469987:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488974720:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788576054.981337:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576055.429628:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576056.005350:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788576056.453395:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576057.029320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576057.477507:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576058.053331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576058.501359:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576059.077342:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788576059.525446:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788576059.589327:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576059.589337:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576059.589341:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576059.589350:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7207 00000100:00080000:0.0:1788576059.589360:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576059.589367:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788576059.589399:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488974848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576059.589431:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576059.589434:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576059.589436:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576059.589440:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7207 00000100:00080000:0.0:1788576059.589443:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576059.589447:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576059.589476:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576059.589490:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488974976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576059.589492:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488974848:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788576059.589507:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576059.589510:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788576059.589512:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576059.589517:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7207 00000100:00080000:0.0:1788576059.589520:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576059.589524:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576059.589538:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576059.589549:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488975104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576059.589559:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576059.589562:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576059.589565:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788576059.589566:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:0.0:1788576059.589571:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7207 00000100:00080000:0.0:1788576059.589575:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576059.589579:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576059.589584:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576059.589594:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488975232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576059.589604:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576059.589608:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788576059.589610:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576059.589614:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7207 00000100:00100000:3.0:1788576059.589614:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576059/real 1788576059] req@ffff94d408d3ca80 x1875450488974848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576059.589620:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576059.589626:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576059.589633:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576059.589643:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488975360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576059.589648:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488974848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576059.589667:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576059.589671:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788576059.589675:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488974848:192.168.202.125@tcp:250:kworker.0 00000100:00100000:3.0:1788576059.589699:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488974976:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788576059.589722:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788576059.589748:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576059/real 1788576059] req@ffff94d408d3ea00 x1875450488974976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576059.589769:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488974976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576059.589777:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576059.589780:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788576059.589782:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488974976:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788576059.589786:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488975104:192.168.202.125@tcp:38:kworker.0 00000800:00000100:3.0:1788576059.589801:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788576059.589809:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576059/real 1788576059] req@ffff94d408d3e680 x1875450488975104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576059.589816:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488975104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576059.589821:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576059.589822:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788576059.589824:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488975104:192.168.202.125@tcp:38:kworker.0 00000100:00100000:3.0:1788576059.589827:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488975232:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788576059.589840:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788576059.589849:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576059/real 1788576059] req@ffff94d408d3c000 x1875450488975232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576059.589859:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488975232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576059.589866:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576059.589890:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788576059.589894:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488975232:192.168.202.125@tcp:8:kworker.0 00000100:00100000:3.0:1788576059.589901:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488975360:192.168.202.125@tcp:8:kworker.0 00000800:00000100:3.0:1788576059.589929:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:3.0:1788576059.589940:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576059/real 1788576059] req@ffff94d408d3d500 x1875450488975360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576059.589952:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488975360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576059.589959:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576059.589972:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:3.0:1788576059.589976:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488975360:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788576060.101358:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576060.549382:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576061.125334:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576061.573352:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576062.149352:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788576062.597371:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576063.173342:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576063.622593:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576064.197872:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576064.645457:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788576064.709462:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576064.709486:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788576064.709490:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576064.709500:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7213 00000100:00080000:3.0:1788576064.709513:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576064.709522:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788576064.709563:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488975488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576064.709644:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488975488:192.168.202.125@tcp:250:kworker.0 00000800:00000100:0.0:1788576064.709708:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:3.0:1788576064.709877:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576064.709882:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576064.709884:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576064.709888:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7213 00000100:00080000:3.0:1788576064.709892:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576064.709896:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576064.709906:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576064.709931:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488975616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576064.710084:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576064/real 1788576064] req@ffff94d408c55f80 x1875450488975488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576064.710105:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488975488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576064.710116:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576064.710120:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576064.710124:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488975488:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788576064.710150:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488975616:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576064.710174:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576064.710199:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576064/real 1788576064] req@ffff94d408c54700 x1875450488975616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576064.710240:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488975616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576064.710681:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576064.710686:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788576064.710688:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576064.710693:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7213 00000100:00080000:3.0:1788576064.710697:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576064.710702:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576064.710708:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576064.710722:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488975744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576064.710732:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576064.710735:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576064.710737:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576064.710752:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7213 00000100:00080000:3.0:1788576064.710756:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576064.710760:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576064.710764:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576064.710776:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488975872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576064.710785:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576064.710788:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788576064.710790:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576064.710793:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7213 00000100:00080000:3.0:1788576064.710796:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576064.710800:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576064.710804:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576064.710813:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488976000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576064.711036:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576064.711041:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576064.711045:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488975616:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576064.711062:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488975744:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576064.711091:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576064.711104:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576064/real 1788576064] req@ffff94d408c56300 x1875450488975744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576064.711116:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488975744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576064.711123:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576064.711126:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576064.711129:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488975744:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576064.711134:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488975872:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576064.711152:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576064.711160:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576064/real 1788576064] req@ffff94d408c55880 x1875450488975872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576064.711170:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488975872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576064.711177:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576064.711180:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576064.711183:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488975872:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576064.711187:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488976000:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576064.711234:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576064.711244:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576064/real 1788576064] req@ffff94d408c55180 x1875450488976000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576064.711527:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488976000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576064.711534:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576064.711537:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576064.711540:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488976000:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788576065.221323:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788576065.669419:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576066.245317:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576066.693418:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576067.269428:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576067.717450:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576068.293521:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788576068.742788:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576069.317491:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576069.765536:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788576069.831620:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576069.831632:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788576069.831637:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576069.831647:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7218 00000100:00080000:3.0:1788576069.831660:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576069.831670:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788576069.831711:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54000 x1875450488976128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576069.831749:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576069.831752:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576069.831754:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576069.831757:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7218 00000100:00080000:3.0:1788576069.831761:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576069.831765:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576069.831774:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576069.831785:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57480 x1875450488976256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576069.831795:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576069.831799:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788576069.831800:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576069.831804:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7218 00000100:00080000:3.0:1788576069.831807:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576069.831809:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576069.831814:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576069.831826:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488976384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576069.831834:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576069.831836:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576069.831838:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576069.831841:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7218 00000100:00080000:3.0:1788576069.831843:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576069.831846:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576069.831849:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576069.831856:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488976512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576069.831863:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576069.831866:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788576069.831868:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576069.831871:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7218 00000100:00080000:3.0:1788576069.831873:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576069.831876:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576069.831880:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576069.831886:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488976640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576069.831986:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488976128:192.168.202.125@tcp:250:kworker.0 00000800:00000100:0.0:1788576069.832042:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576069.832074:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576069/real 1788576069] req@ffff94d408c54000 x1875450488976128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576069.832088:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54000 x1875450488976128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576069.832097:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576069.832100:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576069.832104:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488976128:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788576069.832110:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488976256:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576069.832128:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576069.832138:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576069/real 1788576069] req@ffff94d408c57480 x1875450488976256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576069.832155:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57480 x1875450488976256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576069.832161:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576069.832164:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576069.832167:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488976256:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576069.832171:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488976384:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576069.832186:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576069.832194:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576069/real 1788576069] req@ffff94d408c55180 x1875450488976384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576069.832242:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488976384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576069.832249:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576069.832252:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576069.832255:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488976384:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576069.832260:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488976512:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576069.832277:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576069.832286:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576069/real 1788576069] req@ffff94d408c55880 x1875450488976512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576069.832298:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488976512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576069.832309:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576069.832312:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576069.832314:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488976512:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576069.832319:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488976640:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576069.832334:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576069.832343:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576069/real 1788576069] req@ffff94d408c56300 x1875450488976640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576069.832352:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488976640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576069.832358:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576069.832360:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576069.832364:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488976640:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788576070.342161:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576070.789545:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576071.367045:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788576071.813461:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576072.389774:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576072.837528:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576073.413414:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576073.861660:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576074.437737:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576074.885679:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788576074.949418:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576074.949432:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788576074.949437:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576074.949451:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7223 00000100:00080000:3.0:1788576074.949465:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576074.949474:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788576074.949533:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488976768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576074.949571:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576074.949575:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576074.949577:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576074.949581:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7223 00000100:00080000:3.0:1788576074.949585:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576074.949589:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576074.949600:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576074.949613:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488976896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576074.949641:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576074.949646:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788576074.949648:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576074.949652:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7223 00000100:00080000:3.0:1788576074.949659:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576074.949663:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576074.949668:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576074.949678:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488977024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576074.949687:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576074.949690:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576074.949692:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576074.949695:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7223 00000100:00080000:3.0:1788576074.949697:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576074.949700:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576074.949704:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576074.949712:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488977152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576074.949719:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576074.949722:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788576074.949724:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576074.949726:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7223 00000100:00080000:3.0:1788576074.949729:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576074.949733:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576074.949736:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576074.949743:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488977280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576074.950249:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488976768:192.168.202.125@tcp:250:kworker.0 00000800:00000100:0.0:1788576074.950324:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576074.950368:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576074/real 1788576074] req@ffff94d408c54700 x1875450488976768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576074.950388:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488976768/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576074.950404:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576074.950420:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576074.950424:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488976768:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788576074.950433:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488976896:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576074.950457:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576074.950470:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576074/real 1788576074] req@ffff94d408c55f80 x1875450488976896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576074.950481:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488976896/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576074.950488:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576074.950505:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576074.950509:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488976896:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576074.950515:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488977024:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576074.950538:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576074.950550:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576074/real 1788576074] req@ffff94d408c57800 x1875450488977024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576074.950563:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488977024/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576074.950570:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576074.950573:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576074.950575:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488977024:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576074.950580:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488977152:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576074.950613:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576074.950629:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576074/real 1788576074] req@ffff94d408c56a00 x1875450488977152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576074.950642:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488977152/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576074.950649:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576074.950652:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576074.950655:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488977152:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576074.950662:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488977280:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576074.950685:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576074.950702:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576074/real 1788576074] req@ffff94d408c54380 x1875450488977280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576074.950725:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488977280/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576074.950733:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576074.950737:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576074.950741:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488977280:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788576075.461380:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576075.909452:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576076.485331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576076.933379:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576077.509320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576077.957346:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576078.533764:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576078.981460:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576079.557386:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576080.006335:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788576080.069528:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576080.069539:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576080.069544:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576080.069555:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7228 00000100:00080000:0.0:1788576080.069566:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576080.069586:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788576080.069625:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f100 x1875450488977408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576080.069738:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576080.069743:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576080.069745:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576080.069750:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7228 00000100:00080000:0.0:1788576080.069754:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576080.069759:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576080.069768:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576080.069782:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f800 x1875450488977536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576080.069792:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576080.069796:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788576080.069797:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576080.069801:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7228 00000100:00080000:0.0:1788576080.069804:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576080.069808:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576080.069812:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576080.069838:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488977664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576080.069848:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576080.069851:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576080.069853:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576080.069857:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7228 00000100:00080000:0.0:1788576080.069860:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576080.069864:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576080.069869:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576080.069879:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488977792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576080.069889:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576080.069892:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788576080.069894:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576080.069897:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7228 00000100:00080000:0.0:1788576080.069901:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576080.069904:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576080.069909:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576080.069919:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488977920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788576080.070311:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488977408:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788576080.070396:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488977536:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788576080.070422:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488977664:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788576080.070442:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488977792:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788576080.070461:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488977920:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576080.581839:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576081.029470:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576081.605394:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576082.053443:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576082.629297:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576083.078545:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788576083.141926:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:2.0:1788576083.141968:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788576083.141984:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788576083.142025:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788576083.142034:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788576083.142041:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:2.0:1788576083.142048:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:1.0:1788576083.142187:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576080/real 1788576083] req@ffff94d408d3f100 x1875450488977408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576083.142252:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f100 x1875450488977408/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576083.142265:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576083.142269:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576083.142274:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488977408:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788576083.142281:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576080/real 1788576083] req@ffff94d408d3f800 x1875450488977536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576083.142294:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f800 x1875450488977536/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576083.142302:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576083.142306:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576083.142309:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488977536:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788576083.142316:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576080/real 1788576083] req@ffff94d408d3d500 x1875450488977664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576083.142328:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488977664/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576083.142336:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576083.142339:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576083.142342:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488977664:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788576083.142349:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576080/real 1788576083] req@ffff94d408d3c000 x1875450488977792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576083.142360:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488977792/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576083.142366:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576083.142369:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576083.142372:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488977792:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788576083.142377:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576080/real 1788576083] req@ffff94d408d3e680 x1875450488977920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576083.142387:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488977920/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576083.142393:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576083.142396:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576083.142399:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488977920:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576083.653319:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576084.101699:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576084.677327:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576085.125422:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788576085.190651:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576085.190661:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576085.190665:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576085.190674:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7233 00000100:00080000:2.0:1788576085.190684:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576085.190692:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788576085.190724:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63100 x1875450488978048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576085.190753:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576085.190757:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576085.190759:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576085.190763:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7233 00000100:00080000:2.0:1788576085.190767:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576085.190770:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576085.190779:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576085.190792:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60e00 x1875450488978176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576085.190802:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576085.190806:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788576085.190808:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576085.190811:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7233 00000100:00080000:2.0:1788576085.190814:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576085.190817:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576085.190822:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576085.190832:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488978304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576085.190841:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576085.190844:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576085.190846:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576085.190849:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7233 00000100:00080000:2.0:1788576085.190853:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576085.190856:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576085.190861:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576085.190872:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488978432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576085.190881:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576085.190884:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788576085.190887:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576085.190890:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7233 00000100:00080000:2.0:1788576085.190903:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576085.190908:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576085.190914:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576085.190923:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488978560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788576085.191008:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488978048:192.168.202.125@tcp:250:kworker.0 00000800:00000100:1.0:1788576085.191065:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576085.191095:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576085/real 1788576085] req@ffff94d408c63100 x1875450488978048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576085.191112:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63100 x1875450488978048/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576085.191122:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576085.191126:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576085.191129:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488978048:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788576085.191136:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488978176:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788576085.191157:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576085.191170:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576085/real 1788576085] req@ffff94d408c60e00 x1875450488978176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576085.191181:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60e00 x1875450488978176/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576085.191189:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576085.191192:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576085.191195:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488978176:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788576085.191200:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488978304:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788576085.191248:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576085.191260:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576085/real 1788576085] req@ffff94d408c61c00 x1875450488978304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576085.191272:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488978304/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576085.191279:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576085.191282:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576085.191285:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488978304:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788576085.191291:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488978432:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788576085.191311:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576085.191321:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576085/real 1788576085] req@ffff94d408c61880 x1875450488978432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576085.191339:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488978432/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576085.191347:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576085.191350:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576085.191353:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488978432:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788576085.191358:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488978560:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788576085.191376:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576085.191387:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576085/real 1788576085] req@ffff94d408c61500 x1875450488978560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576085.191398:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488978560/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576085.191405:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576085.191408:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576085.191411:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488978560:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576085.701344:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576086.150266:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576086.725450:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576087.173468:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576087.749355:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576088.197390:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576088.773764:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576089.221369:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576089.797365:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576090.245923:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788576090.310631:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576090.310644:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576090.310648:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576090.310671:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7238 00000100:00080000:0.0:1788576090.310689:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576090.310698:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788576090.310742:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488978688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576090.310778:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576090.310782:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576090.310783:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576090.310787:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7238 00000100:00080000:0.0:1788576090.310791:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576090.310795:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576090.310803:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576090.310816:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488978816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576090.310825:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576090.310828:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788576090.310830:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576090.310833:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7238 00000100:00080000:0.0:1788576090.310836:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576090.310839:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576090.310843:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576090.310853:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488978944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576090.310861:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576090.310864:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576090.310866:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576090.310869:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7238 00000100:00080000:0.0:1788576090.310873:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576090.310876:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576090.310881:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576090.310889:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488979072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576090.310897:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576090.310900:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788576090.310901:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576090.310905:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7238 00000100:00080000:0.0:1788576090.310908:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576090.310912:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576090.310916:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576090.310923:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488979200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788576090.311041:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488978688:192.168.202.125@tcp:250:kworker.0 00000800:00000100:1.0:1788576090.311106:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576090.311146:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576090/real 1788576090] req@ffff94d408d3ea00 x1875450488978688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576090.311164:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488978688/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576090.311175:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576090.311179:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576090.311183:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488978688:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788576090.311190:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488978816:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788576090.311248:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576090.311262:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576090/real 1788576090] req@ffff94d408d3ca80 x1875450488978816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576090.311274:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488978816/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576090.311281:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576090.311285:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576090.311288:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488978816:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788576090.311294:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488978944:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788576090.311314:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576090.311324:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576090/real 1788576090] req@ffff94d408d3d180 x1875450488978944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576090.311335:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488978944/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576090.311343:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576090.311346:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576090.311350:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488978944:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788576090.311356:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488979072:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788576090.311434:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576090.311445:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576090/real 1788576090] req@ffff94d408d3c700 x1875450488979072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576090.311455:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488979072/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576090.311462:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576090.311466:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576090.311469:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488979072:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788576090.311474:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488979200:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788576090.311494:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576090.311504:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576090/real 1788576090] req@ffff94d408d3df80 x1875450488979200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576090.311515:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488979200/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576090.311521:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576090.311524:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576090.311526:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488979200:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576090.825749:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576091.273807:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576091.845372:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576092.295955:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576092.869341:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576093.317498:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576093.893501:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576094.342386:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576094.917342:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576095.365600:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788576095.429498:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576095.429508:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576095.429512:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576095.429522:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7243 00000100:00080000:0.0:1788576095.429533:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576095.429541:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788576095.429601:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488979328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576095.430363:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576095.430367:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576095.430370:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576095.430375:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7243 00000100:00080000:0.0:1788576095.430380:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576095.430385:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576095.430395:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576095.430409:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488979456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576095.430420:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576095.430423:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788576095.430425:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576095.430428:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7243 00000100:00080000:0.0:1788576095.430432:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576095.430435:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576095.430440:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576095.430451:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ed80 x1875450488979584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576095.430459:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576095.430462:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576095.430464:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576095.430467:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7243 00000100:00080000:0.0:1788576095.430470:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576095.430474:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576095.430479:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576095.430491:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488979712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576095.430501:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576095.430504:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788576095.430505:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576095.431043:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7243 00000100:00080000:0.0:1788576095.431049:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576095.431053:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576095.431058:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576095.431072:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488979840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788576095.432890:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488979328:192.168.202.125@tcp:250:kworker.0 00000800:00000100:1.0:1788576095.432960:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576095.432999:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576095/real 1788576095] req@ffff94d408d3ce00 x1875450488979328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576095.433018:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488979328/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576095.433028:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576095.433337:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576095.433342:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488979328:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788576095.433350:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488979456:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788576095.433377:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576095.433391:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576095/real 1788576095] req@ffff94d408d3f480 x1875450488979456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576095.433404:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488979456/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576095.433413:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576095.433416:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576095.433419:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488979456:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788576095.433425:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488979584:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788576095.433443:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576095.433456:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576095/real 1788576095] req@ffff94d408d3ed80 x1875450488979584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576095.433466:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ed80 x1875450488979584/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576095.433479:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576095.433482:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576095.433485:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488979584:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788576095.433491:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488979712:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788576095.433511:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576095.433521:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576095/real 1788576095] req@ffff94d408d3c380 x1875450488979712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576095.433530:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488979712/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576095.433537:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576095.433539:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576095.433543:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488979712:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788576095.433560:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488979840:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788576095.433579:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576095.433590:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576095/real 1788576095] req@ffff94d408d3d880 x1875450488979840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576095.433602:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488979840/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576095.433608:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576095.433611:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576095.433614:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488979840:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576095.944683:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576096.389478:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576096.965326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576097.413408:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576097.990842:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576098.437602:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576099.013959:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576099.461537:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576100.039862:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576100.486153:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788576100.549432:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576100.549443:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788576100.549446:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576100.549456:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7248 00000100:00080000:1.0:1788576100.549468:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576100.549475:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788576100.549510:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488979968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576100.549541:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576100.549545:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788576100.549547:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576100.549551:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7248 00000100:00080000:1.0:1788576100.549554:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576100.549581:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788576100.549590:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576100.549602:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488980096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576100.549611:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576100.549614:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788576100.549616:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576100.549619:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7248 00000100:00080000:1.0:1788576100.549624:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576100.549627:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788576100.549631:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576100.549641:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488980224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576100.549649:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576100.549652:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788576100.549653:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576100.549657:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7248 00000100:00080000:1.0:1788576100.549660:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576100.549663:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788576100.549667:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576100.549676:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488980352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576100.549685:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576100.549688:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788576100.549689:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576100.549693:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7248 00000100:00080000:1.0:1788576100.549695:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576100.549698:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788576100.549702:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576100.549711:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488980480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576100.549886:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488979968:192.168.202.125@tcp:250:kworker.0 00000800:00000100:0.0:1788576100.549941:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576100.549974:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576100/real 1788576100] req@ffff94d4049bf100 x1875450488979968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576100.549991:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488979968/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576100.550000:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576100.550005:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576100.550008:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488979968:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788576100.550015:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488980096:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576100.550036:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576100.550048:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576100/real 1788576100] req@ffff94d4049be300 x1875450488980096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576100.550059:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488980096/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576100.550067:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576100.550070:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576100.550073:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488980096:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576100.550080:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488980224:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576100.550099:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576100.550109:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576100/real 1788576100] req@ffff94d4049bf800 x1875450488980224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576100.550120:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488980224/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576100.550128:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576100.550131:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576100.550135:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488980224:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576100.550141:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488980352:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576100.550160:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576100.550170:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576100/real 1788576100] req@ffff94d4049bed80 x1875450488980352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576100.550182:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488980352/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576100.550189:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576100.550193:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576100.550196:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488980352:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576100.550223:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488980480:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576100.550242:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576100.550253:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576100/real 1788576100] req@ffff94d4049bc700 x1875450488980480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576100.550264:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488980480/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576100.550271:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576100.550274:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576100.550277:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488980480:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576101.061511:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576101.509409:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576102.085515:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576102.533442:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576103.109360:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576103.557421:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576104.133325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576104.581538:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576105.159434:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576105.605598:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788576105.674820:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576105.674834:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576105.674841:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576105.674853:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7253 00000100:00080000:0.0:1788576105.674867:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576105.674876:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788576105.674922:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3dc00 x1875450488980608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576105.674959:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576105.674963:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576105.674965:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576105.674968:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7253 00000100:00080000:0.0:1788576105.674971:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576105.674975:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576105.674986:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576105.675004:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e300 x1875450488980736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576105.675013:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576105.675017:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788576105.675019:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576105.675024:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7253 00000100:00080000:0.0:1788576105.675028:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576105.675032:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576105.675039:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576105.675052:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d880 x1875450488980864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576105.675061:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576105.675064:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576105.675066:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576105.675070:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7253 00000100:00080000:0.0:1788576105.675073:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576105.675077:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576105.675083:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576105.675093:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c380 x1875450488980992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576105.675101:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576105.675105:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788576105.675106:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576105.675110:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7253 00000100:00080000:0.0:1788576105.675113:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576105.675116:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576105.675121:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576105.675130:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ed80 x1875450488981120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788576105.675243:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488980608:192.168.202.125@tcp:250:kworker.0 00000800:00000100:1.0:1788576105.675324:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576105.675369:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576105/real 1788576105] req@ffff94d408d3dc00 x1875450488980608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576105.675388:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3dc00 x1875450488980608/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576105.675399:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576105.675403:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576105.675407:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488980608:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788576105.675415:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488980736:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788576105.675439:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576105.675452:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576105/real 1788576105] req@ffff94d408d3e300 x1875450488980736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576105.675464:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e300 x1875450488980736/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576105.675471:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576105.675474:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576105.675477:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488980736:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788576105.675483:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488980864:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788576105.675503:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576105.675514:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576105/real 1788576105] req@ffff94d408d3d880 x1875450488980864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576105.675525:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d880 x1875450488980864/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576105.675532:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576105.675536:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576105.675539:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488980864:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788576105.675545:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488980992:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788576105.675564:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576105.675586:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576105/real 1788576105] req@ffff94d408d3c380 x1875450488980992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576105.675598:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c380 x1875450488980992/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576105.675605:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576105.675608:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576105.675611:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488980992:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788576105.675617:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488981120:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788576105.675637:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576105.675647:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576105/real 1788576105] req@ffff94d408d3ed80 x1875450488981120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576105.675663:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ed80 x1875450488981120/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576105.675670:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576105.675673:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576105.675676:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488981120:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576106.182800:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576106.630636:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576107.205365:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576107.654476:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576108.229462:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576108.677632:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576109.253920:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576109.703695:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576110.277326:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576110.725666:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788576110.789808:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576110.789816:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576110.789820:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576110.789830:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7259 00000100:00080000:2.0:1788576110.789839:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576110.789847:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788576110.789887:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488981248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576110.789915:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576110.789919:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576110.789920:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576110.789924:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7259 00000100:00080000:2.0:1788576110.789929:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576110.789934:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576110.789949:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576110.789965:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488981376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576110.789980:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576110.789984:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788576110.789986:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576110.789990:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7259 00000100:00080000:2.0:1788576110.789994:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576110.789997:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576110.790001:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576110.790011:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488981504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576110.790019:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576110.790022:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576110.790024:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576110.790027:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7259 00000100:00080000:2.0:1788576110.790031:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576110.790035:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576110.790039:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576110.790049:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488981632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576110.790058:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576110.790061:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788576110.790063:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576110.790066:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7259 00000100:00080000:2.0:1788576110.790069:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576110.790073:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576110.790078:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576110.790086:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488981760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576110.790177:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488981248:192.168.202.125@tcp:250:kworker.0 00000800:00000100:0.0:1788576110.790261:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576110.790294:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576110/real 1788576110] req@ffff94d408c60a80 x1875450488981248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576110.790311:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488981248/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576110.790320:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576110.790324:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576110.790328:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488981248:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788576110.790336:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488981376:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576110.790359:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576110.790373:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576110/real 1788576110] req@ffff94d408c63b80 x1875450488981376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576110.790386:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488981376/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576110.790393:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576110.790396:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576110.790400:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488981376:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576110.790406:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488981504:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576110.790423:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576110.790434:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576110/real 1788576110] req@ffff94d408c62a00 x1875450488981504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576110.790446:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488981504/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576110.790455:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576110.790459:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576110.790461:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488981504:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576110.790465:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488981632:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576110.790481:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576110.790489:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576110/real 1788576110] req@ffff94d408c61f80 x1875450488981632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576110.790499:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488981632/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576110.790505:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576110.790508:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576110.790511:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488981632:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576110.790516:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488981760:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576110.790533:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576110.790543:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576110/real 1788576110] req@ffff94d408c63480 x1875450488981760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576110.790553:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488981760/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576110.790560:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576110.790562:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576110.790565:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488981760:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576111.301322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576111.749645:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576112.325340:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576112.776404:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576113.349467:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576113.797463:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576114.373304:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576114.821832:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576115.397315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576115.845436:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788576115.909359:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576115.909369:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788576115.909374:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576115.909385:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7264 00000100:00080000:1.0:1788576115.909395:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576115.909401:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788576115.909436:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488981888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576115.909455:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576115.909457:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788576115.909458:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576115.909461:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7264 00000100:00080000:1.0:1788576115.909463:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576115.909465:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788576115.909472:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576115.909479:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488982016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576115.909486:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576115.909488:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788576115.909490:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576115.909492:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7264 00000100:00080000:1.0:1788576115.909494:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576115.909496:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788576115.909499:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576115.909506:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488982144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576115.909509:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488981888:192.168.202.125@tcp:250:kworker.0 00000100:00080000:1.0:1788576115.909512:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576115.909518:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788576115.909519:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576115.909521:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7264 00000100:00080000:1.0:1788576115.909523:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576115.909525:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788576115.909528:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576115.909533:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488982272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576115.909538:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576115.909541:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788576115.909542:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576115.909545:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7264 00000100:00080000:1.0:1788576115.909547:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576115.909549:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788576115.909551:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576115.909557:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488982400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788576115.909576:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576115.909627:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576115/real 1788576115] req@ffff94d4049bc000 x1875450488981888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576115.909646:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488981888/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576115.909656:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576115.909661:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576115.909665:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488981888:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788576115.909675:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488982016:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576115.909697:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576115.909711:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576115/real 1788576115] req@ffff94d4049bfb80 x1875450488982016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576115.909723:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488982016/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576115.909731:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576115.909734:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576115.909737:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488982016:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576115.909760:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488982144:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576115.909780:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576115.909792:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576115/real 1788576115] req@ffff94d4049bd180 x1875450488982144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576115.909805:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488982144/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576115.909813:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576115.909816:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576115.909820:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488982144:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576115.909825:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488982272:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576115.909844:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576115.909855:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576115/real 1788576115] req@ffff94d4049bce00 x1875450488982272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576115.909866:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488982272/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576115.909874:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576115.909877:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576115.909880:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488982272:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576115.909885:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488982400:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576115.909902:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576115.909912:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576115/real 1788576115] req@ffff94d4049bea00 x1875450488982400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576115.909922:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488982400/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576115.909929:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576115.909932:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576115.909935:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488982400:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576116.421350:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576116.869505:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576117.445342:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576117.893347:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576118.469315:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576118.919041:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576119.493429:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576119.941424:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576120.517316:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576120.965455:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788576121.029424:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576121.029435:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576121.029440:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576121.029450:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7269 00000100:00080000:2.0:1788576121.029461:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576121.029469:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788576121.029506:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60700 x1875450488982528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576121.029536:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576121.029540:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576121.029542:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576121.029546:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7269 00000100:00080000:2.0:1788576121.029550:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576121.029553:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576121.029562:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576121.029574:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488982656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576121.029583:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576121.029589:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788576121.029590:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576121.029594:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7269 00000100:00080000:2.0:1788576121.029597:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576121.029601:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576121.029605:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576121.029627:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488982784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576121.029636:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576121.029639:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576121.029641:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576121.029645:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7269 00000100:00080000:2.0:1788576121.029647:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576121.029651:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576121.029655:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576121.029664:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60380 x1875450488982912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576121.029673:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576121.029676:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788576121.029677:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576121.029680:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7269 00000100:00080000:2.0:1788576121.029683:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576121.029686:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576121.029690:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576121.029698:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488983040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576121.029770:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488982528:192.168.202.125@tcp:250:kworker.0 00000800:00000100:0.0:1788576121.029829:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576121.029863:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576121/real 1788576121] req@ffff94d408c60700 x1875450488982528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576121.029880:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60700 x1875450488982528/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576121.029890:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576121.029894:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576121.029898:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488982528:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788576121.029904:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488982656:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576121.029924:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576121.029935:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576121/real 1788576121] req@ffff94d408c62d80 x1875450488982656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576121.029946:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488982656/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576121.029954:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576121.029957:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576121.029960:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488982656:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576121.029965:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488982784:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576121.029981:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576121.029991:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576121/real 1788576121] req@ffff94d408c60000 x1875450488982784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576121.030009:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488982784/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576121.030017:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576121.030019:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576121.030022:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488982784:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576121.030028:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488982912:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576121.030044:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576121.030053:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576121/real 1788576121] req@ffff94d408c60380 x1875450488982912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576121.030065:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60380 x1875450488982912/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576121.030072:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576121.030075:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576121.030078:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488982912:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576121.030084:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488983040:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576121.030100:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576121.030109:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576121/real 1788576121] req@ffff94d408c61180 x1875450488983040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576121.030120:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488983040/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576121.030134:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576121.030137:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576121.030140:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488983040:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576121.542867:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576121.989408:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576122.565345:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576123.013469:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576123.589365:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576124.038884:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576124.613507:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576125.061432:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576125.637414:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576126.085458:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788576126.149407:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576126.149420:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576126.149425:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576126.149448:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7274 00000100:00080000:0.0:1788576126.149463:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576126.149472:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788576126.149522:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f480 x1875450488983168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576126.149571:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576126.149576:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576126.149578:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576126.149582:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7274 00000100:00080000:0.0:1788576126.149590:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576126.149594:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576126.149605:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576126.149618:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ce00 x1875450488983296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576126.149645:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576126.149648:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788576126.149650:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576126.149654:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7274 00000100:00080000:0.0:1788576126.149658:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576126.149662:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576126.149667:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576126.149685:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3df80 x1875450488983424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576126.149695:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576126.149698:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576126.149700:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576126.149704:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7274 00000100:00080000:0.0:1788576126.149707:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576126.149711:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576126.149716:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576126.149725:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c700 x1875450488983552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576126.149733:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576126.149736:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788576126.149738:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576126.149742:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7274 00000100:00080000:0.0:1788576126.149745:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576126.149749:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576126.149753:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576126.149761:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d180 x1875450488983680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788576126.150724:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488983168:192.168.202.125@tcp:250:kworker.0 00000800:00000100:1.0:1788576126.150818:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576126.150863:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576126/real 1788576126] req@ffff94d408d3f480 x1875450488983168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576126.150880:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f480 x1875450488983168/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576126.150892:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576126.150896:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576126.150900:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488983168:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788576126.150907:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488983296:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788576126.150928:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576126.150938:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576126/real 1788576126] req@ffff94d408d3ce00 x1875450488983296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576126.150955:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ce00 x1875450488983296/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576126.150963:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576126.150966:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576126.150969:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488983296:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788576126.150974:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488983424:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788576126.150990:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576126.150999:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576126/real 1788576126] req@ffff94d408d3df80 x1875450488983424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576126.151009:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3df80 x1875450488983424/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576126.151016:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576126.151019:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576126.151022:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488983424:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788576126.151028:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488983552:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788576126.151044:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576126.151051:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576126/real 1788576126] req@ffff94d408d3c700 x1875450488983552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576126.151061:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c700 x1875450488983552/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576126.151072:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576126.151075:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576126.151078:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488983552:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788576126.151083:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488983680:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788576126.151098:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576126.151106:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576126/real 1788576126] req@ffff94d408d3d180 x1875450488983680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576126.151116:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d180 x1875450488983680/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576126.151123:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576126.151126:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576126.151129:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488983680:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576126.661346:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576127.109604:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576127.685402:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576128.133955:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576128.709278:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576129.157942:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576129.733355:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576130.181406:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576130.759541:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576131.209364:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788576131.273472:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576131.273481:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576131.273485:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576131.273494:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7279 00000100:00080000:0.0:1788576131.273505:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576131.273513:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788576131.273552:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ca80 x1875450488983808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576131.273577:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576131.273581:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576131.273583:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576131.273586:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7279 00000100:00080000:0.0:1788576131.273590:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576131.273595:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576131.273604:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576131.273622:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3ea00 x1875450488983936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788576131.273622:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488983808:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788576131.273668:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576131.273672:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788576131.273674:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576131.273679:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7279 00000100:00080000:0.0:1788576131.273691:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576131.273696:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576131.273701:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788576131.273702:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576131.273717:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488984064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576131.273728:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576131.273731:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576131.273734:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788576131.273736:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576131/real 1788576131] req@ffff94d408d3ca80 x1875450488983808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576131.273741:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7279 00000100:00080000:0.0:1788576131.273748:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576131.273753:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788576131.273757:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ca80 x1875450488983808/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788576131.273760:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788576131.273774:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788576131.273777:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488984192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576131.273779:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576131.273785:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488983808:192.168.202.125@tcp:250:kworker.0 00000100:00080000:0.0:1788576131.273802:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576131.273805:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788576131.273808:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788576131.273810:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488983936:192.168.202.125@tcp:38:kworker.0 00000100:00080000:0.0:1788576131.273815:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7279 00000100:00080000:0.0:1788576131.273820:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576131.273825:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000800:00000100:1.0:1788576131.273829:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000008:00100000:0.0:1788576131.273831:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576131.273838:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576131/real 1788576131] req@ffff94d408d3ea00 x1875450488983936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576131.273847:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488984320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788576131.273850:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3ea00 x1875450488983936/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576131.273872:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576131.273877:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576131.273881:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488983936:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788576131.273887:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488984064:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788576131.273908:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576131.273921:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576131/real 1788576131] req@ffff94d408d3e680 x1875450488984064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576131.273933:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488984064/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576131.273942:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576131.273944:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576131.273947:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488984064:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788576131.273953:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488984192:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788576131.273989:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576131.274000:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576131/real 1788576131] req@ffff94d408d3c000 x1875450488984192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576131.274012:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488984192/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576131.274019:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576131.274022:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576131.274025:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488984192:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788576131.274047:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488984320:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788576131.274068:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576131.274079:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576131/real 1788576131] req@ffff94d408d3d500 x1875450488984320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576131.274096:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488984320/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576131.274103:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576131.274106:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576131.274109:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488984320:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576131.782420:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576132.229719:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576132.805324:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576133.253501:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576133.829311:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576134.278695:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576134.853339:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576135.301702:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576135.877331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576136.325402:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788576136.389638:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576136.389648:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788576136.389663:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576136.389674:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7284 00000100:00080000:1.0:1788576136.389684:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576136.389692:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788576136.389729:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488984448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576136.389806:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488984448:192.168.202.125@tcp:250:kworker.0 00000800:00000100:0.0:1788576136.389866:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:1.0:1788576136.389899:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576136.389903:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788576136.389905:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576136.389910:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7284 00000100:00080000:1.0:1788576136.389914:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576136.389919:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788576136.389927:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576136.389941:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488984576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576136.390105:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576136/real 1788576136] req@ffff94d4049bc380 x1875450488984448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576136.390124:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc380 x1875450488984448/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576136.390133:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576136.390137:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576136.390141:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488984448:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788576136.390164:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488984576:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576136.390188:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576136.390223:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576136/real 1788576136] req@ffff94d4049bf480 x1875450488984576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576136.390237:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488984576/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576136.390245:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576136.390249:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576136.390252:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488984576:192.168.202.125@tcp:38:kworker.0 00000100:00080000:1.0:1788576136.390352:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576136.390356:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788576136.390358:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576136.390362:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7284 00000100:00080000:1.0:1788576136.390366:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576136.390370:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788576136.390376:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576136.390390:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf480 x1875450488984704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576136.390399:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576136.390401:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788576136.390403:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576136.390406:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7284 00000100:00080000:1.0:1788576136.390409:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576136.390412:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788576136.390416:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576136.390423:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc380 x1875450488984832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576136.390431:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576136.390434:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788576136.390435:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576136.390438:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7284 00000100:00080000:1.0:1788576136.390441:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576136.390444:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788576136.390448:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576136.390456:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bea00 x1875450488984960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576136.390515:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488984704:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576136.390540:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576136.390553:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576136/real 1788576136] req@ffff94d4049bf480 x1875450488984704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576136.390565:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf480 x1875450488984704/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576136.390573:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576136.390576:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576136.390631:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488984704:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576136.390639:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488984832:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576136.390703:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576136.390716:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576136/real 1788576136] req@ffff94d4049bc380 x1875450488984832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576136.390729:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc380 x1875450488984832/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576136.390736:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576136.390739:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576136.390742:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488984832:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576136.390746:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488984960:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576136.390761:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576136.390768:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576136/real 1788576136] req@ffff94d4049bea00 x1875450488984960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576136.390774:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bea00 x1875450488984960/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576136.390779:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576136.390781:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576136.390783:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488984960:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576136.901288:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576137.349572:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576137.925329:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576138.375229:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576138.949331:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576139.397440:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576139.973387:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576140.421558:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576140.997356:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576141.445492:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788576141.509326:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576141.509337:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788576141.509342:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576141.509352:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7289 00000100:00080000:1.0:1788576141.509362:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576141.509371:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788576141.509408:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bce00 x1875450488985088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576141.509441:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576141.509444:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788576141.509446:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576141.509451:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7289 00000100:00080000:1.0:1788576141.509454:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576141.509458:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788576141.509467:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576141.509480:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd180 x1875450488985216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576141.509489:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576141.509492:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788576141.509493:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576141.509496:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7289 00000100:00080000:1.0:1788576141.509499:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576141.509503:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788576141.509507:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576141.509516:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bfb80 x1875450488985344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576141.509526:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576141.509528:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788576141.509530:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576141.509534:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7289 00000100:00080000:1.0:1788576141.509537:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576141.509540:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788576141.509544:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576141.509554:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc000 x1875450488985472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576141.509561:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576141.509564:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788576141.509566:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576141.509569:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7289 00000100:00080000:1.0:1788576141.509572:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576141.509575:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788576141.509579:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576141.509588:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bc700 x1875450488985600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576141.509679:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488985088:192.168.202.125@tcp:250:kworker.0 00000800:00000100:0.0:1788576141.509739:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576141.509774:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576141/real 1788576141] req@ffff94d4049bce00 x1875450488985088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576141.509791:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bce00 x1875450488985088/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576141.509800:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576141.509803:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576141.509807:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488985088:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788576141.509813:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488985216:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576141.509833:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576141.509844:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576141/real 1788576141] req@ffff94d4049bd180 x1875450488985216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576141.509855:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd180 x1875450488985216/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576141.509863:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576141.509865:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576141.509868:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488985216:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576141.509874:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488985344:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576141.509889:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576141.509899:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576141/real 1788576141] req@ffff94d4049bfb80 x1875450488985344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576141.509917:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bfb80 x1875450488985344/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576141.509925:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576141.509927:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576141.509930:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488985344:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576141.509935:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488985472:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576141.509952:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576141.509961:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576141/real 1788576141] req@ffff94d4049bc000 x1875450488985472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576141.509971:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc000 x1875450488985472/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576141.509978:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576141.509981:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576141.509984:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488985472:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576141.509989:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488985600:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576141.510008:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576141.510018:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576141/real 1788576141] req@ffff94d4049bc700 x1875450488985600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576141.510028:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bc700 x1875450488985600/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576141.510041:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576141.510044:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576141.510047:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488985600:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576142.021320:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576142.470013:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576143.045385:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576143.495119:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576144.069389:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576144.519570:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576145.094223:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576145.541438:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576146.117402:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576146.565445:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788576146.629312:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576146.629325:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788576146.629331:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576146.629341:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7294 00000100:00080000:1.0:1788576146.629688:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576146.629697:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788576146.629730:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bed80 x1875450488985728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576146.629770:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576146.629774:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788576146.629776:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576146.629781:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7294 00000100:00080000:1.0:1788576146.629785:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576146.629789:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788576146.629798:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576146.629812:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf800 x1875450488985856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576146.629822:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576146.629825:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788576146.629827:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576146.629831:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7294 00000100:00080000:1.0:1788576146.629834:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576146.629838:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788576146.629842:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576146.629853:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be300 x1875450488985984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576146.629868:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576146.629871:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788576146.629873:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576146.629877:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7294 00000100:00080000:1.0:1788576146.629880:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576146.629884:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788576146.629889:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576146.629914:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bf100 x1875450488986112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576146.629922:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576146.629926:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788576146.629928:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576146.629931:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7294 00000100:00080000:1.0:1788576146.629935:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576146.629938:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788576146.629943:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576146.629953:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488986240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576146.632487:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488985728:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788576146.632595:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488985856:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576146.632624:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488985984:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576146.632646:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488986112:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576146.632669:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488986240:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576147.141338:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576147.591231:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576148.165339:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576148.615796:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576149.189379:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576149.637460:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788576149.701512:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.125:988 00000400:00000100:1.0:1788576149.701553:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.125@tcp at host 192.168.202.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788576149.701567:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788576149.701609:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788576149.701618:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788576149.701626:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000800:00000100:1.0:1788576149.701633:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.125@tcp 00000100:00100000:0.0:1788576149.704359:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576146/real 1788576149] req@ffff94d4049bed80 x1875450488985728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576149.704396:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bed80 x1875450488985728/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576149.704412:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576149.704416:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576149.704420:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488985728:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788576149.704429:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576146/real 1788576149] req@ffff94d4049bf800 x1875450488985856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576149.704442:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf800 x1875450488985856/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576149.704449:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576149.704453:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576149.704456:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488985856:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576149.704463:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576146/real 1788576149] req@ffff94d4049be300 x1875450488985984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576149.704475:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be300 x1875450488985984/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576149.704483:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576149.704486:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576149.704488:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488985984:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576149.704494:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576146/real 1788576149] req@ffff94d4049bf100 x1875450488986112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576149.704505:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bf100 x1875450488986112/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576149.704512:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576149.704516:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576149.704519:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488986112:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576149.704525:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576146/real 1788576149] req@ffff94d4049bca80 x1875450488986240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576149.704535:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488986240/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576149.704542:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576149.704545:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576149.704547:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488986240:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576150.213362:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576150.662151:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576151.237307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576151.691633:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788576151.754002:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576151.754012:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788576151.754016:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576151.754026:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7300 00000100:00080000:3.0:1788576151.754037:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576151.754044:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788576151.754078:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488986368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576151.754112:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576151.754115:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576151.754117:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576151.754121:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7300 00000100:00080000:3.0:1788576151.754124:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576151.754128:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576151.754136:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576151.754149:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488986496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576151.754159:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576151.754165:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788576151.754166:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576151.754170:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7300 00000100:00080000:3.0:1788576151.754173:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576151.754177:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576151.754181:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576151.754190:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488986624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576151.754199:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576151.754233:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576151.754235:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576151.754238:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7300 00000100:00080000:3.0:1788576151.754241:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576151.754255:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576151.754259:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576151.754269:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488986752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576151.754277:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576151.754280:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788576151.754282:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576151.754285:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7300 00000100:00080000:3.0:1788576151.754287:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576151.754290:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576151.754295:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576151.754302:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488986880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576151.754401:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488986368:192.168.202.125@tcp:250:kworker.0 00000800:00000100:0.0:1788576151.754458:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576151.754491:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576151/real 1788576151] req@ffff94d408c55c00 x1875450488986368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576151.754506:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488986368/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576151.754516:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576151.754520:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576151.754523:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488986368:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788576151.754532:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488986496:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576151.754550:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576151.754560:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576151/real 1788576151] req@ffff94d408c54e00 x1875450488986496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576151.754571:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488986496/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576151.754579:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576151.754581:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576151.754584:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488986496:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576151.754589:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488986624:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576151.754604:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576151.754612:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576151/real 1788576151] req@ffff94d408c57b80 x1875450488986624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576151.754623:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488986624/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576151.754630:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576151.754633:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576151.754636:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488986624:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576151.754642:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488986752:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576151.754656:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576151.754664:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576151/real 1788576151] req@ffff94d408c56d80 x1875450488986752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576151.754676:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488986752/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576151.754683:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576151.754685:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576151.754688:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488986752:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576151.755122:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488986880:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576151.755149:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576151.762461:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576151/real 1788576151] req@ffff94d408c56680 x1875450488986880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576151.762490:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488986880/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576151.762501:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576151.762505:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576151.762509:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488986880:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576152.261830:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576152.710927:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576153.285688:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576153.734267:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576154.310800:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576154.758377:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576155.333325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576155.781742:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576156.368057:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576156.805872:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788576156.869268:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576156.869277:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788576156.869281:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576156.869291:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7305 00000100:00080000:3.0:1788576156.869300:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576156.869307:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788576156.869339:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57100 x1875450488987008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576156.869369:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576156.869373:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576156.869375:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576156.869378:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7305 00000100:00080000:3.0:1788576156.869381:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576156.869384:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576156.869391:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576156.869401:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54a80 x1875450488987136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576156.869410:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576156.869413:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788576156.869415:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576156.869418:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7305 00000100:00080000:3.0:1788576156.869421:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576156.869424:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576156.869428:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576156.869437:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56680 x1875450488987264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576156.869444:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576156.869447:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576156.869448:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576156.869451:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7305 00000100:00080000:3.0:1788576156.869454:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576156.869456:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576156.869460:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576156.869466:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56d80 x1875450488987392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576156.869474:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576156.869476:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788576156.869478:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576156.869480:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7305 00000100:00080000:3.0:1788576156.869483:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576156.869485:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576156.869488:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576156.869494:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57b80 x1875450488987520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576156.869573:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488987008:192.168.202.125@tcp:250:kworker.0 00000800:00000100:0.0:1788576156.869636:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576156.869666:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576156/real 1788576156] req@ffff94d408c57100 x1875450488987008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576156.869682:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57100 x1875450488987008/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576156.869690:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576156.869694:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576156.869697:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488987008:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788576156.869713:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488987136:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576156.869734:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576156.869745:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576156/real 1788576156] req@ffff94d408c54a80 x1875450488987136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576156.869755:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54a80 x1875450488987136/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576156.869762:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576156.869765:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576156.869767:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488987136:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576156.869772:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488987264:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576156.869788:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576156.869798:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576156/real 1788576156] req@ffff94d408c56680 x1875450488987264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576156.869809:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56680 x1875450488987264/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576156.869822:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576156.869825:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576156.869827:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488987264:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576156.869833:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488987392:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576156.869848:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576156.869857:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576156/real 1788576156] req@ffff94d408c56d80 x1875450488987392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576156.869868:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56d80 x1875450488987392/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576156.869875:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576156.869878:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576156.869881:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488987392:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576156.869886:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488987520:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576156.869903:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576156.869912:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576156/real 1788576156] req@ffff94d408c57b80 x1875450488987520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576156.869922:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57b80 x1875450488987520/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576156.869928:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576156.869931:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576156.869935:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488987520:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576157.381483:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576157.829611:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576158.405357:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576158.857059:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576159.430059:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576159.878330:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576160.453409:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576160.901546:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576161.477900:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576161.925441:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788576161.989471:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576161.989485:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788576161.989491:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576161.989503:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7310 00000100:00080000:1.0:1788576161.989518:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576161.989650:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:1.0:1788576161.989699:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd880 x1875450488987648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576161.989742:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576161.989747:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788576161.989749:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576161.989753:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7310 00000100:00080000:1.0:1788576161.989758:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576161.989762:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788576161.989773:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576161.989787:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049be680 x1875450488987776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576161.989797:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576161.989800:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788576161.989802:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576161.989805:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7310 00000100:00080000:1.0:1788576161.989809:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576161.989812:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788576161.989816:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576161.989825:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bdc00 x1875450488987904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576161.989843:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576161.989846:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788576161.989848:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576161.989852:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7310 00000100:00080000:1.0:1788576161.989855:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576161.989859:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788576161.989865:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576161.989875:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bd500 x1875450488988032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576161.989889:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576161.989892:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788576161.989894:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576161.989899:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7310 00000100:00080000:1.0:1788576161.989902:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576161.989906:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:1.0:1788576161.989911:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576161.989923:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d4049bca80 x1875450488988160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576161.992836:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488987648:192.168.202.125@tcp:250:kworker.0 00000800:00000100:0.0:1788576161.992933:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576161.992986:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576161/real 1788576161] req@ffff94d4049bd880 x1875450488987648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576161.993007:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd880 x1875450488987648/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576161.993021:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576161.993026:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576161.993030:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488987648:192.168.202.125@tcp:250:kworker.0 00000100:00100000:0.0:1788576161.993039:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488987776:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576161.993061:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576161.993072:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576161/real 1788576161] req@ffff94d4049be680 x1875450488987776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576161.993084:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049be680 x1875450488987776/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576161.993093:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576161.993097:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576161.993100:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488987776:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576161.993107:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488987904:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576161.993126:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576161.993137:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576161/real 1788576161] req@ffff94d4049bdc00 x1875450488987904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576161.993148:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bdc00 x1875450488987904/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576161.993156:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576161.993160:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576161.993163:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488987904:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576161.993170:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488988032:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576161.993190:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576161.993200:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576161/real 1788576161] req@ffff94d4049bd500 x1875450488988032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576161.993288:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bd500 x1875450488988032/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576161.993297:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576161.993300:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576161.993303:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488988032:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576161.993310:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488988160:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576161.993333:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576161.993343:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576161/real 1788576161] req@ffff94d4049bca80 x1875450488988160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576161.993354:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d4049bca80 x1875450488988160/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576161.993362:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576161.993365:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576161.993368:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d4049bca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488988160:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576162.501911:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576162.953837:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576163.525388:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576163.983834:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576164.549385:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576164.997931:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576165.573295:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576166.022193:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576166.597889:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576167.045477:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788576167.109382:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576167.109396:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576167.109401:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576167.109414:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7315 00000100:00080000:2.0:1788576167.109428:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576167.109437:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788576167.109486:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62300 x1875450488988288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576167.109517:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576167.109520:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576167.109522:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576167.109526:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7315 00000100:00080000:2.0:1788576167.109530:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576167.109533:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576167.109544:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576167.109557:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62680 x1875450488988416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576167.109567:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788576167.109578:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488988288:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788576167.109659:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:0.0:1788576167.109664:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788576167.109664:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576167.109673:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7315 00000100:00080000:2.0:1788576167.109678:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576167.109683:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576167.109689:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576167.109703:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63800 x1875450488988544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576167.109713:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576167.109717:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1788576167.109719:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576167/real 1788576167] req@ffff94d408c62300 x1875450488988288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576167.109720:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576167.109726:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7315 00000100:00080000:2.0:1788576167.109729:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576167.109749:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576167.109757:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576167.109765:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62300 x1875450488988288/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576167.109774:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61180 x1875450488988672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576167.109780:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576167.109786:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576167.109789:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00080000:2.0:1788576167.109793:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1788576167.109797:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488988288:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788576167.109798:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576167.109807:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7315 00000100:00080000:2.0:1788576167.109812:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576167.109817:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788576167.109833:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488988416:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576167.109859:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576167.109873:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576167/real 1788576167] req@ffff94d408c62680 x1875450488988416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576167.109885:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62680 x1875450488988416/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576167.109894:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576167.109897:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576167.109901:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488988416:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576167.109906:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488988544:192.168.202.125@tcp:38:kworker.0 00000800:00000100:0.0:1788576167.109935:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576167.109946:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576167/real 1788576167] req@ffff94d408c63800 x1875450488988544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576167.109958:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63800 x1875450488988544/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576167.109966:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576167.109970:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576167.109973:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488988544:192.168.202.125@tcp:38:kworker.0 00000100:00100000:0.0:1788576167.109979:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488988672:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576167.109996:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576167.110005:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576167/real 1788576167] req@ffff94d408c61180 x1875450488988672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576167.110017:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61180 x1875450488988672/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576167.110025:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576167.110028:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576167.110032:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488988672:192.168.202.125@tcp:8:kworker.0 00000008:00100000:2.0:1788576167.110236:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576167.110253:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60380 x1875450488988800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576167.110310:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488988800:192.168.202.125@tcp:8:kworker.0 00000800:00000100:0.0:1788576167.110335:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:0.0:1788576167.110348:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576167/real 1788576167] req@ffff94d408c60380 x1875450488988800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576167.110360:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60380 x1875450488988800/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576167.110368:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576167.110372:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:0.0:1788576167.110375:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488988800:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576167.621343:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576168.071266:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576168.645364:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576169.094175:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576169.669307:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576170.119060:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576170.699886:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576171.150084:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576171.717886:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576172.165592:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788576172.235231:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576172.235250:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576172.235255:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576172.235269:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7320 00000100:00080000:0.0:1788576172.235285:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576172.235296:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:0.0:1788576172.235349:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f800 x1875450488988928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576172.235396:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576172.235400:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576172.235402:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576172.235407:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7320 00000100:00080000:0.0:1788576172.235411:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576172.235415:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576172.235427:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576172.235441:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3f100 x1875450488989056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576172.235452:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576172.235455:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788576172.235457:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576172.235461:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7320 00000100:00080000:0.0:1788576172.235465:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576172.235470:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576172.235475:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576172.235489:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3d500 x1875450488989184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576172.235498:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576172.235502:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576172.235504:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576172.235508:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7320 00000100:00080000:0.0:1788576172.235511:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576172.235515:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576172.235520:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576172.235530:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3c000 x1875450488989312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576172.235538:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576172.235542:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788576172.235543:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576172.235547:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7320 00000100:00080000:0.0:1788576172.235550:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576172.235553:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:0.0:1788576172.235557:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576172.235567:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408d3e680 x1875450488989440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788576172.235702:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488988928:192.168.202.125@tcp:250:kworker.0 00000800:00000100:1.0:1788576172.235797:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576172.235843:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576172/real 1788576172] req@ffff94d408d3f800 x1875450488988928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576172.236023:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f800 x1875450488988928/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576172.236034:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576172.236038:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576172.236041:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488988928:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788576172.236049:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488989056:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788576172.236074:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576172.236087:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576172/real 1788576172] req@ffff94d408d3f100 x1875450488989056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576172.236099:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3f100 x1875450488989056/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576172.236106:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576172.236109:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576172.236111:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488989056:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788576172.236116:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488989184:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788576172.236134:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576172.236145:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576172/real 1788576172] req@ffff94d408d3d500 x1875450488989184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576172.236156:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3d500 x1875450488989184/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576172.236162:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576172.236165:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576172.236168:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488989184:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788576172.236173:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488989312:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788576172.236192:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576172.236243:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576172/real 1788576172] req@ffff94d408d3c000 x1875450488989312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576172.236256:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3c000 x1875450488989312/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576172.236264:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576172.236267:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576172.236270:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488989312:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788576172.236276:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488989440:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788576172.236298:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576172.236310:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576172/real 1788576172] req@ffff94d408d3e680 x1875450488989440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576172.236320:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408d3e680 x1875450488989440/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576172.236328:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576172.236331:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576172.236335:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408d3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488989440:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576172.741372:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576173.189561:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576173.765394:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576174.213593:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576174.789553:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576175.237413:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576175.813281:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576176.261488:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576176.837690:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576177.287990:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788576177.349479:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576177.349496:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576177.349502:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576177.349515:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7325 00000100:00080000:2.0:1788576177.349529:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576177.349538:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788576177.349598:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60000 x1875450488989568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576177.349640:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576177.349644:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576177.349647:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576177.349659:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7325 00000100:00080000:2.0:1788576177.349664:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576177.349669:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576177.349679:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576177.349692:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62d80 x1875450488989696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576177.349701:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576177.349705:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788576177.349707:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576177.349711:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7325 00000100:00080000:2.0:1788576177.349714:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576177.349718:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576177.349723:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576177.349733:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60700 x1875450488989824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576177.349743:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576177.349746:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576177.349748:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576177.349751:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7325 00000100:00080000:2.0:1788576177.349755:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576177.349758:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576177.349763:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576177.349773:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63480 x1875450488989952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576177.349782:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576177.349785:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788576177.349786:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576177.349790:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7325 00000100:00080000:2.0:1788576177.349794:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576177.349797:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576177.349801:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576177.349812:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61f80 x1875450488990080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788576177.350750:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488989568:192.168.202.125@tcp:250:kworker.0 00000800:00000100:1.0:1788576177.350810:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576177.350843:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576177/real 1788576177] req@ffff94d408c60000 x1875450488989568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576177.350860:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60000 x1875450488989568/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576177.350875:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576177.350878:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576177.350882:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488989568:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788576177.350889:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488989696:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788576177.350906:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576177.350917:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576177/real 1788576177] req@ffff94d408c62d80 x1875450488989696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576177.350927:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62d80 x1875450488989696/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576177.350935:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576177.350937:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576177.350940:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488989696:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788576177.350945:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488989824:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788576177.350959:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576177.350968:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576177/real 1788576177] req@ffff94d408c60700 x1875450488989824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576177.350978:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60700 x1875450488989824/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576177.350986:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576177.350988:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576177.350991:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488989824:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788576177.350996:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488989952:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788576177.351010:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576177.351019:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576177/real 1788576177] req@ffff94d408c63480 x1875450488989952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576177.351029:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63480 x1875450488989952/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576177.351036:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576177.351039:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576177.351042:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488989952:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788576177.351046:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488990080:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788576177.351060:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576177.351068:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576177/real 1788576177] req@ffff94d408c61f80 x1875450488990080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576177.351078:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61f80 x1875450488990080/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576177.351086:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576177.351268:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576177.351274:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488990080:192.168.202.125@tcp:8:kworker.0 00000100:00100000:0.0:1788576177.861357:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576178.313493:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576178.885353:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788576179.334007:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576179.909506:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576180.357672:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576180.939555:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576181.382595:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576181.957347:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788576182.405839:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788576182.469778:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576182.469789:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576182.469794:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576182.469805:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7330 00000100:00080000:2.0:1788576182.469816:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576182.469824:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788576182.469865:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c62a00 x1875450488990208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576182.469904:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576182.469909:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576182.469911:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576182.469915:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7330 00000100:00080000:2.0:1788576182.469918:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576182.469923:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576182.469933:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576182.469948:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63b80 x1875450488990336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788576182.469951:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488990208:192.168.202.125@tcp:250:kworker.0 00000100:00080000:2.0:1788576182.469970:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576182.469974:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788576182.469976:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576182.469980:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7330 00000100:00080000:2.0:1788576182.469985:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576182.469989:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576182.469995:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576182.470010:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60a80 x1875450488990464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576182.470019:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576182.470023:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576182.470025:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576182.470028:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7330 00000100:00080000:2.0:1788576182.470032:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788576182.470036:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00080000:2.0:1788576182.470039:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576182.470045:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576182.470059:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488990592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576182.470069:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576182.470072:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788576182.470074:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576182.470080:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7330 00000100:00100000:1.0:1788576182.470081:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576182/real 1788576182] req@ffff94d408c62a00 x1875450488990208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576182.470085:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576182.470093:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576182.470101:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576182.470108:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c62a00 x1875450488990208/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576182.470114:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488990720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576182.470123:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576182.470130:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576182.470149:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c62a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488990208:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788576182.470180:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488990336:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788576182.470246:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576182.470261:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576182/real 1788576182] req@ffff94d408c63b80 x1875450488990336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576182.470275:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63b80 x1875450488990336/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576182.470286:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576182.470289:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576182.470293:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488990336:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788576182.470299:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488990464:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788576182.470319:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576182.470330:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576182/real 1788576182] req@ffff94d408c60a80 x1875450488990464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576182.470349:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60a80 x1875450488990464/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576182.470358:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576182.470361:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576182.470365:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488990464:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788576182.470370:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488990592:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788576182.470390:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576182.470400:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576182/real 1788576182] req@ffff94d408c61500 x1875450488990592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576182.470411:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488990592/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576182.470419:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576182.470421:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576182.470424:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488990592:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788576182.470430:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488990720:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788576182.470446:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576182.470455:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576182/real 1788576182] req@ffff94d408c61880 x1875450488990720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576182.470465:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488990720/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576182.470492:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576182.470496:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576182.470500:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488990720:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788576182.981370:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576183.432904:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576184.005495:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576184.453567:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576185.029332:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788576185.477370:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576186.053426:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576186.501655:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576187.077362:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576187.525490:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788576187.589716:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576187.589724:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788576187.589734:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576187.589742:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7335 00000100:00080000:3.0:1788576187.589750:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576187.589755:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788576187.589784:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54e00 x1875450488990848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576187.589806:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576187.589809:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576187.589810:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576187.589813:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7335 00000100:00080000:3.0:1788576187.589814:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576187.589816:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576187.589822:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576187.589831:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55c00 x1875450488990976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576187.589840:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576187.589843:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788576187.589844:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576187.589848:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7335 00000100:00080000:3.0:1788576187.589851:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576187.589855:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576187.589859:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576187.589868:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54380 x1875450488991104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576187.589877:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576187.589880:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576187.589881:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576187.589885:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7335 00000100:00080000:3.0:1788576187.589887:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576187.589890:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576187.589893:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576187.589899:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56a00 x1875450488991232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576187.589904:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576187.589906:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788576187.589907:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576187.589909:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7335 00000100:00080000:3.0:1788576187.589911:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576187.589912:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576187.589914:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576187.589918:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c57800 x1875450488991360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788576187.590110:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488990848:192.168.202.125@tcp:250:kworker.0 00000800:00000100:1.0:1788576187.590179:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576187.590239:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576187/real 1788576187] req@ffff94d408c54e00 x1875450488990848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576187.590257:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54e00 x1875450488990848/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576187.590278:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576187.590282:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576187.590286:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488990848:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788576187.590293:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488990976:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788576187.590315:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576187.590327:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576187/real 1788576187] req@ffff94d408c55c00 x1875450488990976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576187.590344:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55c00 x1875450488990976/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576187.590351:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576187.590355:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576187.590358:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488990976:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788576187.590364:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488991104:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788576187.590383:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576187.590394:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576187/real 1788576187] req@ffff94d408c54380 x1875450488991104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576187.590405:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54380 x1875450488991104/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576187.590412:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576187.590416:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576187.590419:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488991104:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788576187.590425:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488991232:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788576187.590443:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576187.590453:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576187/real 1788576187] req@ffff94d408c56a00 x1875450488991232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576187.590465:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56a00 x1875450488991232/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576187.590472:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576187.590475:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576187.590478:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488991232:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788576187.590483:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488991360:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788576187.590500:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576187.590510:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576187/real 1788576187] req@ffff94d408c57800 x1875450488991360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576187.590520:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c57800 x1875450488991360/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576187.590526:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576187.590529:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576187.590532:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488991360:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788576188.101325:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:3.0:1788576188.551330:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576189.125602:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576189.573553:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576190.149417:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576190.603374:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576191.173660:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576191.623836:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576192.200266:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576192.645546:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788576192.709331:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576192.709343:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788576192.709348:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576192.709359:0:51031:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7341 00000100:00080000:3.0:1788576192.709369:0:51031:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576192.709376:0:51031:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:3.0:1788576192.709419:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55f80 x1875450488991488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576192.709451:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576192.709455:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576192.709457:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576192.709460:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7341 00000100:00080000:3.0:1788576192.709464:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576192.709468:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576192.709478:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576192.709513:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c54700 x1875450488991616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576192.709524:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576192.709527:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788576192.709529:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576192.709532:0:51031:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7341 00000100:00080000:3.0:1788576192.709536:0:51031:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576192.709540:0:51031:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576192.709545:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576192.709557:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c56300 x1875450488991744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576192.709566:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576192.709569:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576192.709570:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576192.709574:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7341 00000100:00080000:3.0:1788576192.709577:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576192.709581:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576192.709586:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576192.709596:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55880 x1875450488991872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576192.709606:0:51031:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576192.709610:0:51031:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788576192.709611:0:51031:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576192.709616:0:51031:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7341 00000100:00080000:3.0:1788576192.709618:0:51031:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576192.709622:0:51031:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:3.0:1788576192.709625:0:51031:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576192.709635:0:51031:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c55180 x1875450488992000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788576192.709692:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488991488:192.168.202.125@tcp:250:kworker.0 00000800:00000100:1.0:1788576192.709794:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576192.709849:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576192/real 1788576192] req@ffff94d408c55f80 x1875450488991488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576192.709869:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55f80 x1875450488991488/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576192.709881:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576192.709886:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576192.709890:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488991488:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788576192.709898:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488991616:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788576192.709924:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576192.709937:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576192/real 1788576192] req@ffff94d408c54700 x1875450488991616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576192.709950:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c54700 x1875450488991616/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576192.709959:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576192.709963:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576192.709966:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c54700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488991616:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788576192.709973:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488991744:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788576192.709994:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576192.710006:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576192/real 1788576192] req@ffff94d408c56300 x1875450488991744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576192.710018:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c56300 x1875450488991744/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576192.710026:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576192.710029:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576192.710032:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488991744:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788576192.710038:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488991872:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788576192.710060:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576192.710071:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576192/real 1788576192] req@ffff94d408c55880 x1875450488991872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576192.710083:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55880 x1875450488991872/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576192.710098:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576192.710102:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576192.710105:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488991872:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788576192.710112:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488992000:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788576192.710151:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576192.710163:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576192/real 1788576192] req@ffff94d408c55180 x1875450488992000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576192.710174:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c55180 x1875450488992000/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576192.710182:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576192.710185:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576192.710188:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488992000:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788576193.226744:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576193.669430:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576194.246555:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576194.694138:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576195.269534:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576195.717687:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576196.336692:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576196.741553:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576197.383890:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576197.766183:0:70972:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788576197.829465:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2a9ed50a-a289-4e84-96b9-fb493e8051f0->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576197.829474:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576197.829479:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576197.829489:0:36550:0:(import.c:510:import_select_connection()) MGC192.168.202.125@tcp: connect to NID 192.168.202.125@tcp last attempt 7346 00000100:00080000:2.0:1788576197.829499:0:36550:0:(import.c:589:import_select_connection()) MGC192.168.202.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576197.829507:0:36550:0:(import.c:624:import_select_connection()) MGC192.168.202.125@tcp: import ffff94d405b33800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000100:00100000:2.0:1788576197.829543:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61c00 x1875450488992128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576197.829573:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576197.829576:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576197.829578:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576197.829582:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7346 00000100:00080000:2.0:1788576197.829585:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576197.829589:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff94d402e39800: import ffff94d411203000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576197.829598:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576197.829610:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c60e00 x1875450488992256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576197.829620:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576197.829623:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788576197.829625:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576197.829628:0:36550:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7346 00000100:00080000:2.0:1788576197.829631:0:36550:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576197.829635:0:36550:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff94d402e39800: import ffff94d405c94000 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576197.829639:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683137 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576197.829649:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c63100 x1875450488992384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576197.829658:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576197.829661:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576197.829663:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576197.829666:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7346 00000100:00080000:2.0:1788576197.829669:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576197.829672:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff94d402e39800: import ffff94d405c96800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576197.829676:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576197.829685:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61880 x1875450488992512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576197.829693:0:36550:0:(pinger.c:229:ptlrpc_pinger_process_import()) 86f37580-247f-4c91-a842-1b0c4422007b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576197.829696:0:36550:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788576197.829698:0:36550:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576197.829701:0:36550:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: connect to NID 192.168.202.125@tcp last attempt 7346 00000100:00080000:2.0:1788576197.829703:0:36550:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576197.829706:0:36550:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff94d402e39800: import ffff94d405c93800 using connection 192.168.202.125@tcp/192.168.202.125@tcp 00000008:00100000:2.0:1788576197.829710:0:36550:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683137 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576197.829718:0:36550:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff94d408c61500 x1875450488992640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788576197.829793:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488992128:192.168.202.125@tcp:250:kworker.0 00000800:00000100:1.0:1788576197.829849:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576197.829883:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576197/real 1788576197] req@ffff94d408c61c00 x1875450488992128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576197.829914:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61c00 x1875450488992128/t0(0) o250->MGC192.168.202.125@tcp@192.168.202.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576197.829931:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405b33800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576197.829934:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576197.829938:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a9ed50a-a289-4e84-96b9-fb493e8051f0:2402:1875450488992128:192.168.202.125@tcp:250:kworker.0 00000100:00100000:1.0:1788576197.829945:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488992256:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788576197.829966:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576197.829978:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576197/real 1788576197] req@ffff94d408c60e00 x1875450488992256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576197.829989:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c60e00 x1875450488992256/t0(0) o38->lustre-MDT0000-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576197.829997:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d411203000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576197.830000:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576197.830003:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c60e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488992256:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788576197.830009:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488992384:192.168.202.125@tcp:38:kworker.0 00000800:00000100:1.0:1788576197.830039:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576197.830050:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576197/real 1788576197] req@ffff94d408c63100 x1875450488992384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576197.830061:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c63100 x1875450488992384/t0(0) o38->lustre-MDT0001-mdc-ffff94d402e39800@192.168.202.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576197.830070:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c94000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576197.830073:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576197.830076:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c63100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488992384:192.168.202.125@tcp:38:kworker.0 00000100:00100000:1.0:1788576197.830083:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488992512:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788576197.830100:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576197.830110:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576197/real 1788576197] req@ffff94d408c61880 x1875450488992512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576197.830121:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61880 x1875450488992512/t0(0) o8->lustre-OST0000-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576197.830128:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c96800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576197.830131:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576197.830134:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488992512:192.168.202.125@tcp:8:kworker.0 00000100:00100000:1.0:1788576197.830139:0:2402:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488992640:192.168.202.125@tcp:8:kworker.0 00000800:00000100:1.0:1788576197.830157:0:2402:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.125@tcp 00000100:00100000:1.0:1788576197.830167:0:2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576197/real 1788576197] req@ffff94d408c61500 x1875450488992640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576197.830178:0:2402:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff94d408c61500 x1875450488992640/t0(0) o8->lustre-OST0001-osc-ffff94d402e39800@192.168.202.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576197.830185:0:2402:0:(import.c:65:import_set_state_nolock()) ffff94d405c93800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576197.830188:0:2402:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.125@tcp failed (-110) 00000100:00100000:1.0:1788576197.830191:0:2402:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff94d408c61500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:86f37580-247f-4c91-a842-1b0c4422007b:2402:1875450488992640:192.168.202.125@tcp:8:kworker.0 00000100:00100000:2.0:1788576198.405322:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:0.0:1788576198.791609:0:71181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576199.446757:0:71063:0:(client.c:2702:ptlrpc_set_wait()) set ffff94d408235780 going to sleep for 0 seconds 00010000:00010000:1.0:1788576199.823615:0:70973:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) Debug log: 78172 lines, 78172 kept, 0 dropped, 0 bad.